Most on-device LLMs fail the same way: they’re confidently wrong and have no idea. Cactus Hybrid fixes the “no idea” half. It’s a post-trained version of Gemma 4 E2B with confidence probes baked directly into the checkpoint — every answer comes back with a score from 0 to 1 as structured data, so your app knows exactly how much to trust it.
What it actually is
This is a model, not an app. Cactus took Gemma 4 E2B — the smallest of the family — and trained it to know when it’s out of its depth. You set a threshold (say 0.85): above it, the answer runs fully on-device; below it, your code routes the query to a cloud model — Claude, GPT, Gemini, your pick. The kicker: this tiny model matches Gemini 3.1 Flash-Lite on most benchmarks while sending only 15–35% of queries to the cloud. The phone handles the easy 80%, the cloud takes the hard 20%, and the user never sees the seam.
Why it’s worth watching
On-device inference has always had a reliability problem, and “just use a bigger model” defeats the point of running local. Confidence-as-structured-data is the clean answer — routing becomes a number comparison, not a guess. It landed as a Show HN hit for exactly that reason.
The SDK ships copy-paste quickstarts for Cactus, Transformers, llama.cpp and MLX, and the score comes back as structured data you can branch on directly. Typical use case: a mobile assistant that answers instantly offline for routine stuff and quietly escalates only when it’s unsure — cutting cloud bills and latency without the usual local-model embarrassment.
You Might Also Like
- Gemini 3 1 Flash Lite Googles Cheapest Model Just got Surprisingly Good
- Hypura Runs a 31gb Model on a 32gb mac at 2 2 tok s Llama cpp Just Ooms
- Googles Gemini 3 1 Flash Live Scores 90 8 on Audio Benchmarks Real Time Voice ai Gets Serious
- Ollama mlx on Apple Silicon 1810 Tokens sec Prefill and the end of Llama cpp on mac
- Gemini 3 1 Flash Lite Hits ga 0 25 m Input Tokens 2 5x Faster Ttft
Top AI Product 
Leave a comment