PrismML just shipped something that would have sounded absurd two years ago. Bonsai 27B is a quantized version of Qwen3.6-27B — not a new model, just a compressed one. The 1-bit variant runs at 3.9GB. The ternary variant, which uses three weight values instead of two, runs at 5.9GB. Both are Apache 2.0 licensed, and both will fit on a laptop with room to spare (MarkTechPost, July 2026).
The practical upshot for Python developers is straightforward. You can now run a model with 27 billion parameters locally — not through an API, not through a cloud GPU rental, but on the machine you already own. If you’ve been building LLM-powered Python applications and paying per-token inference costs, the math changed last week.
What Bonsai 27B actually is
Bonsai 27B is not a new pretrain. The architecture is Qwen3.6-27B, unchanged. What PrismML did was compress the weights into extremely low-bit representations while preserving as much of the original model’s capability as possible.
Two variants shipped side by side. The ternary build uses three weight values: −1, 0, and +1, at an effective 1.71 bits per weight. This compresses the 54GB FP16 model down to 5.9GB. The 1-bit build goes further, using only binary weights (−1, +1) at 1.125 bits per weight, for a 3.9GB footprint.
For context, the standard 4-bit quantized version of Qwen3.6-27B (Q4_K_XL) takes 17.6GB and requires a high-end GPU. The ternary Bonsai runs on a machine with 8GB of RAM. The 1-bit build runs on a phone.
The model is multimodal. The full parameter count breaks down as roughly 24.8 billion for language, 460 million for vision, and 2.5 billion for embeddings and the language model head. The vision tower is stored separately at 4-bit precision using HQQ (Half-Quadratic Quantization). This split design is intentional — the vision tower degrades more aggressively under extreme quantization than the language tower, so PrismML kept it at a higher precision. The result is a model that can process images and text together while still fitting in a fraction of the original footprint.
PrismML identifies four deployment patterns for the model. Laptop-local agents use the ternary build for full-repository code tasks over the full 262K context. Phone-local reasoning runs the 1-bit build for on-device privacy-sensitive workflows. Combined with a 4-bit KV cache, a single 24GB GPU can serve the ternary model to multiple users. And for fully offline or air-gapped environments, both builds work without any network connectivity whatsoever.
How much quality you lose
The answer depends on the task, and in some cases it’s less than you’d expect.
On math benchmarks, the FP16 original scores 95.33. The ternary Bonsai scores 93.40. The 1-bit build scores 91.66. That’s a surprisingly small drop from 16-bit floating point to literal binary weights. Math appears to be relatively robust to extreme quantization.
On coding tasks, the FP16 baseline scores 88.74. The ternary build hits 85.96. The 1-bit version drops to 81.88. Still functional for code completion and explanation, but you’ll notice the gap on complex multi-file reasoning.
The thinking benchmark — which measures the model’s ability to reason through multi-step problems — tells a slightly different story. FP16 scores 85.07. A standard 4-bit quant (Q4_K_XL) actually scores 84.99, nearly identical. The ternary Bonsai drops to 80.49. The 1-bit build falls to 76.11. Thinking is where low-bit quantization hurts most, likely because reasoning chains are sensitive to small perturbations in attention patterns that binary weights introduce.
On knowledge and reasoning, agentic tool calling, and instruction following, the degradation follows the same pattern: ternary is usable for most tasks, 1-bit is usable for some. The ternary build sits between 4-bit and 2-bit quantization in quality while being smaller than either. The 1-bit build gives you a 27B-class model at a 3.9GB footprint at the cost of non-trivial quality loss.
For Python developers building applications on top of these models, the decision tree is: if you need the highest quality and have a GPU, use FP16 or 4-bit. If you’re building something that runs locally for privacy or cost reasons and can tolerate a modest quality drop, the ternary Bonsai is a compelling option. If you need a model on-device for a mobile application and 76 on thinking benchmarks is acceptable, the 1-bit build opens doors that were previously closed.
One thing the benchmarks don’t capture is latency. A 27B model running locally on a laptop responds faster than any API call to a cloud-hosted model, regardless of token generation speed, because the round-trip network latency is zero. For interactive applications — chatbots, code completions, real-time document Q&A — this matters more than raw tokens-per-second. The ternary Bonsai on an M3 Max should feel snappier than GPT-4o over an API, even if the output quality is lower.
Running Bonsai 27B from Python
PrismML ships the model in GGUF format for llama.cpp and MLX format for Apple Silicon. Both are accessible from Python.
For llama.cpp, the server setup is a single command:
./scripts/start_llama_server.sh
This spins up an OpenAI-compatible API on port 8080 with chat and vision UI. From Python, you talk to it the same way you’d talk to any OpenAI-compatible endpoint:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
response = client.chat.completions.create(
model="ternary-bonsai-27b",
messages=[{"role": "user", "content": "Explain KV cache growth"}]
)
For Apple Silicon, the MLX integration is even simpler:
mlx_lm.generate --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \
--prompt "Explain KV cache growth."
The model supports a 262K token context window. That’s made practical by the fact that roughly 75% of Qwen3.6-27B’s attention layers are linear rather than quadratic, so the memory cost of long contexts grows more slowly than in a standard transformer. For Python applications that process long documents, codebases, or multi-turn conversations, this matters.
The 262K context also enables full-repository code work. The ternary build, at 5.9GB, can fit an entire mid-sized codebase into context on a laptop with 16GB of RAM. That means code review, refactoring suggestions, and cross-file bug analysis without uploading proprietary code to a third-party API. For teams with strict data policies, this is the killer feature — not the model quality, but the fact that the code never leaves the machine.
PrismML also provides a pure-Python path through the MLX library for Apple Silicon users. If you’re on an M-series Mac, you don’t need llama.cpp at all. The entire pipeline — model loading, tokenization, generation — runs through MLX with no C++ dependencies. This lowers the barrier for Python developers who want to experiment with local LLMs but don’t want to manage a separate compilation toolchain.
Performance: faster than you’d think
Compressing weights to 1 bit doesn’t automatically mean fast inference. But PrismML ships a speculative decoding drafter called DSpark that’s trained specifically against the Bonsai target.
On an H100 with draft depth k=4, the binary Bonsai reaches an accepted length of τ=3.6 tokens per draft step. That translates to 143.8 tokens per second — a 1.37× speedup over the undrafted model. Verification is lossless, so the output distribution is identical to what you’d get without the drafter, just faster.
On Apple Silicon, the drafter is disabled by default at batch size 1. The raw inference speed on an M3 Max should still be usable for interactive applications, but the big win here is on server-class hardware where you want to serve multiple users from a single 24GB GPU.
On mobile, PrismML’s whitepaper measures 672 tokens per 1% of iPhone battery for the 1-bit build. That puts multi-thousand-token generation within reach of a phone battery cycle. It’s not fast, but it’s possible — and that’s a threshold that FP16 and even 4-bit quantization can’t cross on mobile hardware.
The density numbers tell the efficiency story in more concrete terms. The FP16 original processes 0.051 “thinking units” per GB — effectively, you need a dedicated GPU to run it at all. The ternary Bonsai processes 0.400 per GB, nearly 8× more efficient per unit of memory. The 1-bit build hits 0.530 per GB. You’re trading a few benchmark points for a massive increase in deployment flexibility.
What this means for Python ML developers
The immediate takeaway for anyone building LLM-powered Python applications: the cost equation for running models locally shifted materially last week.
If you’ve been building a RAG pipeline, a code assistant, or a document analysis tool and paying per-token rates to an API provider, it’s worth benchmarking the ternary Bonsai against your current model on your specific task. The quality gap between ternary Bonsai and the 4-bit quantized version of the same model is small enough that the cost savings of local inference could tip the decision — especially for applications where data privacy or latency matters.
If you’re building mobile applications, the 1-bit build changes the conversation entirely. A 27B model on a phone, even with degraded quality on some benchmarks, is a capability that didn’t exist in any practical form before this release.
If you’re running a small team or a solo project and your monthly API bill has been creeping up, the ternary Bonsai on a single machine could replace several hundred dollars a month in inference costs. At 5.9GB, it runs alongside your development tools without requiring a dedicated GPU server.
The code is open source under Apache 2.0. No commercial restrictions. No usage limits. The model weights, the drafter, and the serving infrastructure are all available on GitHub and Hugging Face. If you know your way around llama.cpp or MLX, you can have this running in an afternoon.
The quantization techniques themselves — PrismML didn’t invent binary-weight LLMs, but they’ve pushed the quality-retention frontier forward — are worth studying if you work on model compression or deployment optimization. The ternary build in particular, at 1.71 bits per weight with math and coding scores within single-digit percentages of FP16, resets the baseline for what extreme quantization can achieve.
One last thing worth noting: the Bonsai release includes both GGUF and MLX format weights, plus the DSpark drafter and the full serving stack. That’s a complete deployment pipeline, not just model weights dumped on Hugging Face. The barrier to trying this on your own hardware is low. Download, configure a single script, and you’ve got an OpenAI-compatible endpoint serving a 27B model on a laptop. Two years ago, that sentence would have been science fiction. Last week, it shipped.
Related reading: Fine-Tuning LLMs on a Single GPU: QLoRA Best Practices for 2026, AI Coding Tools for Data Science and Machine Learning in 2026: What Python Developers Need to Know
Discussion
Leave a comment
No comments yet
Be the first to start the conversation.