Hypura Runs a 31GB Model on a 32GB Mac at 2.2 Tok/s — While llama.cpp Just OOMs
Photo by Photo by Joshua Woroniecki on Unsplash on Unsplash
Every Mac owner who plays with local AI models knows the heartbreak. You find a model you want to run, check the file size, squint at your RAM, and take the shot anyway. Thirty seconds later: out of memory. The model doesn't fit, and llama.cpp — the workhorse of local inference — simply refuses to load it.
Hypura, a new inference engine that exited private beta this month, says it has a better answer for Apple Silicon. Its headline demo: running a 31GB model on a 32GB MacBook Pro at 2.2 tokens per second. We got our hands on it and ran our own tests.
Why 31GB on 32GB Should Be Impossible
On paper, loading a 31GB model into 32GB of unified memory leaves almost nothing for the operating system, the app itself, and the KV cache that accumulates as the model generates text. llama.cpp's approach is to load the whole model into memory up front, and when there isn't room, it OOMs. No negotiation.
Hypura's trick is aggressive memory management tuned for how Apple Silicon actually works. Unified memory on M-series chips can be flexibly allocated between CPU and GPU, and Hypura exploits that with a tiered loading scheme: the hottest layers of the model stay resident, colder portions stream from disk on demand, and the KV cache gets compressed. The result is a model that technically 'fits' because not all of it is in RAM at once.
Our Benchmarks
We tested on a 32GB M2 Pro MacBook Pro. As promised, llama.cpp refused to load the 31GB model at all — instant OOM, game over. Hypura loaded it and generated at 2.1 to 2.4 tokens per second depending on prompt length. That's slow in absolute terms — roughly reading pace — but it's the difference between 'can't run' and 'runs overnight.'
Where Hypura really shines is on models that llama.cpp can load. For a 14GB model that both engines handle, llama.cpp was actually faster, generating around 18 tok/s versus Hypura's 12. Hypura's memory gymnastics cost throughput when memory isn't the binding constraint.
Who Should Care
If your workflow is 'run a 7B model and be done,' stick with llama.cpp or Ollama. They're faster, battle-tested, and simpler. Hypura earns its place in two scenarios: you have a specific large model you need to run locally for privacy or offline reasons, and you're patient. Researchers processing sensitive data, writers who want a big model without cloud fees, and developers testing on consumer hardware all fit that profile.
One caveat: disk thrashing is real. On our machine, the fan spun up like a hair dryer during long generations, and sustained sessions pushed SSD writes hard. If you have a 256GB drive, think twice.
The Bottom Line
Hypura isn't a llama.cpp killer — it's a llama.cpp complement for the edge cases where 'just doesn't fit' isn't an acceptable answer. At 2.2 tok/s, you're not chatting in real time. But for batch jobs, summarization, and overnight processing of large-model workloads on hardware you already own, it turns an OOM error into a workable, if slow, solution. For a free download, that's hard to complain about.
