Top AI Product

18.7k GitHub stars, +448 in a day. KTransformers is an open-source inference framework from Tsinghua’s MADSys lab and Approaching.AI, and the idea behind it is almost rude in its simplicity: a MoE model only activates a few experts per token, so why is the whole thing sitting in VRAM?

So it isn’t. Attention and shared experts stay on the GPU where bandwidth matters. Routed experts get dumped into system RAM and run on AMX-optimized CPU kernels. One RTX 5090 with 32GB VRAM handles 100B+ models; a 671B model hits 286 tokens/s on a single box. The paper landed at SOSP 31.

The part that matters: no quantization tax

Most “run a huge model at home” tricks are just aggressive quantization wearing a costume. KTransformers keeps native BF16 and FP8 per-channel precision. You get the real model, not a lobotomized one.

How you actually use it

pip install ktransformers gives you a Python library plus an OpenAI-compatible server, so it drops in as a local backend behind existing clients. Apache-2.0. Day-0 support for MiniMax-M2.5, GLM-5.2, DeepSeek-V4-Flash.

You Might Also Like

• Mercury 2 Just hit 1000 Tokens per Second and its not Even Using Transformers
• Microsoft Bitnet 100b Parameters on a Single cpu 0 4 gb of Memory Zero Gpus
• Unsloth Studio Brings no Code llm Fine Tuning to Your Laptop 2x Faster 70 Less Vram
• Arm agi cpu 136 Cores 3nm 2x Performance per Rack and 9 Companies Already Signed up
• 0 004 per Task how Atlas Squeezes Frontier Level Coding From a Single 500 gpu