AirLLM runs 70B–2.8T models on a single small GPU by streaming layers

AirLLM is an open-source Python library that slashes the GPU memory needed for large-model inference by loading a model one layer at a time from disk rather than holding the whole thing in VRAM. The result is that a 70B model fits on a single 4GB card with no quantization, distillation, or pruning required, and the same one-line AutoModel.from_pretrained() call scales up to Llama 3.1 405B on 8GB. For sparse mixture-of-experts models the trick goes further: because only the experts a token actually routes to get streamed in, DeepSeek-V3 (671B) runs in roughly 12GB and Kimi K3 (2.8T), billed as the largest open-source model to date, was measured end to end in 3.72GB on one RTX 6000 Ada.

The tradeoff is speed, not accuracy — the bottleneck becomes disk I/O, so AirLLM leans on prefetching to overlap loading with compute and offers optional block-wise 4-bit/8-bit weight-only compression that the project claims can triple throughput with negligible accuracy loss. It works out of the box with most popular open families (Llama, Qwen, DeepSeek, Mistral/Mixtral, Phi, Gemma, ChatGLM, and others), supports CPU inference and Apple Silicon via MLX, and can optionally delete the original weights after transformation to save disk.

The practical significance is accessibility: it lets developers experiment with frontier-scale open models on consumer hardware or cheap cloud instances, trading latency for the ability to run models that would otherwise demand a multi-GPU server. Newer models carry their own baggage, though — Kimi K3 support, for instance, forces flash-attention and pins specific CUDA 12 and transformers 4.56.x builds, a reminder that ‘runs on a single card’ still assumes a fairly particular software stack.