Bonsai 27B
Canonical version: Bonsai 27B.
Bonsai 27B is an extremely low-bit compression of Qwen3.6-27B from PrismML, released on 14 July 2026 under Apache 2.0. The 1-bit variant is 3.9 GB. That is a 27B-class model, with reasoning and vision intact, running on an iPhone.
For context: the full-precision model is about 54 GB, and a conventional 4-bit build is 18 GB.
The two variants
| Weights | Effective bits | Size | Benchmark retention | |
|---|---|---|---|---|
| Ternary | {−1, 0, +1} + FP16 group scaling | 1.71 | 5.9 GB | 95% of baseline (80.5) |
| 1-bit | {−1, +1} + FP16 group scaling | 1.125 | 3.9 GB | 90% of baseline (76.1) |
Retention is measured across 15 benchmarks in thinking mode. Math and coding degrade very little. Tool calling is where it hurts: 74.0 ternary and 66.0 for 1-bit, against an 80.0 baseline. If you plan to use this in an agent loop, that is the number that decides whether it works, not the headline average.
Speed: up to 163 tok/s (1-bit) and 134 tok/s (ternary) on an RTX 5090; 87 and 58 tok/s on an M5 Max.
Why "1-bit" needs an asterisk
A weight is still a bit. The fractional bit counts come from group-wise scaling: a group of weights (128 here) shares one FP16 scale factor, so the per-weight cost averages out to something between 1 and 2 once you amortize the scale.
The ternary variant is not binary at all. Its values are {−1, 0, +1}, which is a trit, not a bit. Five trits pack into a byte at 1.6 bits each, close enough to the theoretical optimum that squeezing further costs more in unpacking time than it saves in memory.
Worth knowing: the current release appears to use llama.cpp's Q2_0 (2 bits per weight plus a scale per group), which wastes one of four available bit patterns on ternary weights. PrismML's fork fixes the group-size mismatch but not the wasted pattern. There is real headroom left.
Why it matters
The compression is trained in, not bolted on. This is the distinction that matters and the one most easily missed. Post-training quantization to 2 bits, of the kind you get from an off-the-shelf UD_Q2 build, lands near the 1-bit numbers here while losing more where it counts. A 5% drop in tool-calling reads as small on a table and shows up constantly in real use. Bonsai's low-bit representation runs end to end, with no higher-precision layers anywhere, including a 4-bit vision tower.
On-device stops meaning small. A 12 GB iPhone has roughly 6 GB usable for a model once you account for KV cache and activations. Until now that ceiling meant models in the single-digit billions. A 27B-class model with a 262K context and multimodal input now fits under it. That is a step change in what "local" means, not an increment.
Compression is becoming its own layer of the stack. PrismML is a Caltech spinout backed by Khosla, Cerberus, Google, and Samsung, and its entire business is compressing networks without losing reasoning. The pattern to watch: labs train frontier models, specialists compress them, and the gap between "what runs in a datacenter" and "what runs in a pocket" is being attacked by a different set of people than the ones training the models.
Caveats
- Tool calling degrades most. Fine for chat, summarization, and reasoning. Test it hard before putting it in an agent
- Tooling had not caught up at release. Reports of the GGUF and MLX builds failing in LM Studio, pending llama.cpp and MLX engine updates
- Running the 27B on iOS via the Locally AI app requires an iPhone 17 Pro or Pro Max
- Benchmarks are PrismML's own, comparing against their own baseline. The relative claim (retains 90-95% of the source model) is the meaningful one, and it is the kind of claim that is easy to verify independently. Worth checking before relying on it
- Speedups from low-bit formats are not automatic. Unpacking ternary encodings costs cycles, and several people report no gain over a straight Q4 quantization of the same base model
References
- Announcement — https://prismml.com/news/bonsai-27b
- Models — https://huggingface.co/prism-ml/models
- Hacker News discussion (706 points) — https://news.ycombinator.com/item?id=48910545
Related
- Qwen3.6-27B
- Qwen
- AI Quantization
- AI Open Weight Models
- Running AI Models Locally
- Large Language Models (LLMs)
- AI Tool Use
- llama.cpp
- MLX
- LM Studio
- Ollama
- AI Foundation Models
- Kimi K3
About Sébastien
Ready to get to the next level?
Found this valuable? Share it with someone who needs it.