Top AI Product

On-device LLMs have been stuck at 3B–8B for years. PrismML, a Caltech spinout backed by Khosla, Google and Samsung, just shipped Bonsai 27B: a Qwen3.6 27B multimodal model crushed down to 5.9GB (ternary) or 3.9GB (1-bit). That 3.9GB number is not an accident — it’s roughly the app memory budget iOS gives you. HN put it on the front page with 266 points.

The compression actually holds up

Across 15 reasoning, coding, math and vision benchmarks, the ternary build averages 80.5 against the full-precision baseline’s 85.0 — about 95% of the quality at 9x less weight. The 1-bit build keeps roughly 90%. It still takes images, still does 262K context, and runs at ~11 tokens/sec on an iPhone 17 Pro. Slow, but it’s a 27B model on a phone.

How you run it

Apache 2.0 open weights, commercial use included. PrismML wrote its own low-bit inference kernels — MLX for Apple silicon, CUDA for NVIDIA — plus GGUF builds for llama.cpp. There’s no hosted API; the shipped path is a local OpenAI-compatible server with tool calling and vision input, so an existing agent stack can point at localhost and keep working. Offline assistants, private vision apps, laptop-grade agents: that’s the target.

You Might Also Like

• Google Turboquant Squeezes llm Cache to 3 Bits 6x Less Memory 8x Faster Zero Accuracy Loss
• Ollama mlx on Apple Silicon 1810 Tokens sec Prefill and the end of Llama cpp on mac
• Prismml Exits Stealth With 16m and a 1 bit Model That Rivals Llama 3 at 1 16th the Memory
• 700 Github Stars in a Week Apfel Exposes the Free llm Apple Locked Behind Siri
• Tinygpu George Hotz got Apple to Sign an Nvidia gpu Driver for mac