Top AI Product

The memory wall for local LLMs just lost another brick. Slotstream is an open-source Swift inference engine from developer carloslfu that runs Qwen3.8-Flash-Next — 125B parameters, 104GB on disk at 4-bit — on a 48GB Mac at ~12 tok/s. It hit No. 1 on Show HN with 238 points and 117 comments.

How an SSD becomes VRAM

The model is MoE: only 10 of 512 experts per layer fire per token. So Slotstream keeps the 3.8GB shared weights in RAM and leaves 68GB of routed experts on SSD, streaming them on demand into a fixed pool of cache slots shared across all 48 layers — layers borrow slots from each other. Peak RAM: 32GB. Startup: ~2 seconds. Even an 8GB Mac runs it; 16GB gets ~4 tok/s.

Drop-in local API

One Swift binary, no Python. It ships Ollama-compatible and OpenAI-compatible endpoints, so existing chat apps and SDKs point at localhost and get a 125B-class coding model, fully offline. The catch: it’s hardcoded to this one model’s geometry, Apple Silicon only.

That specialization is exactly why it beats llama.cpp’s mmap-and-pray approach here.

You Might Also Like

• Ollama mlx on Apple Silicon 1810 Tokens sec Prefill and the end of Llama cpp on mac
• 397 Billion Parameters on a 48gb Macbook Flash moe Turns Apples 2023 Research Into Reality
• Hypura Runs a 31gb Model on a 32gb mac at 2 2 tok s Llama cpp Just Ooms
• Qwen3 8 Flash Next Alibabas Qwen4 Architecture Preview Trains at 1 9 the Cost of Qwen3 7 Plus
• Notebooklm py the Unofficial Python sdk That Finally Gives Google Notebooklm a Real api