Top AI Product

An 80B model normally wants 40GB+ of memory. Swiftlet, a new Show HN project, is a Swift + Metal inference runtime that runs Qwen3-Next / 3.5 / 3.6 MoE models on Apple devices with almost none: 42GB on disk, 4.3GB peak RAM.

The trick: SSD as VRAM

MoE models only activate ~3B parameters per token. Swiftlet keeps the dense core resident in memory and streams routed expert weights from SSD on demand — fixed-stride single reads from a .qpack container, plus an LFU cache for hot experts. 75% of layers use Gated DeltaNet linear attention, so the KV cache stops growing with context. That’s how a 35B model ends up running on an iPhone at ~1 token/s.

A local OpenAI-compatible API

This isn’t just a demo. Swiftlet ships a Swift package, a CLI, and an OpenAI-compatible local server — point any existing OpenAI client at localhost and you get free, private inference. There’s also Priv AI, an App Store app built on it.

1 token/s won’t replace your API bills. But this is the most aggressive memory squeeze in local inference since airllm — and “what runs on a phone” just jumped to 35B.

You Might Also Like

• Turbofieldfare Gemma 4 26b Running in 2gb ram on any m Series mac
• Cursor Composer 2 Takes on Anthropic and Openai With a 0 50 m Token Coding Model and the Benchmarks Back it up
• Openai Desktop Superapp Chatgpt Codex and Atlas are Becoming one app
• Embarrassingly Simple Self Distillation ssd Boosts Qwen3 30b Code Scores by 30 no Teachers no rl no Tricks
• Ring app Store Amazon Opens its 100m Camera Fleet to Third Party Apps