AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) — the largest open-source model released to date — on under 4GB, because sparse MoE models stream one expert at a time rather than a whole layer.

Updates

[2026/08] Qwen3.8-27B support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in 3.33GB of VRAM, measured end to end on one RTX 3090. Needs transformers 5.8+.

[2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: pip install compressed-tensors flash-attn (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformers 4.56.x, as its remote code does not load on 5.x.

[2026/06] v3.0: FP8 model support + the latest models. Run DeepSeek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel.

[2024/08/20] v2.11.0: Support Qwen2.5

.

.

Looks like Updates happened after 2 years. Sharing this for AirLLM fans.