On-device runtime runs 80B Qwen in 4.3 GB by streaming MoE experts from SSD

Swiftlet is a Swift and Metal runtime that runs large Qwen3-Next and Qwen3.5/3.6 mixture-of-experts models on Apple Silicon by keeping only each model’s small dense core resident in memory and fetching the routed expert weights from SSD as they’re needed. The payoff is a dramatic cut in RAM: an 80B model that occupies 42 GB on disk runs in about 4.3 GB, and a 35B model runs on an iPhone 17 in roughly 2.5 GB — which the author claims is the first time a model of this class has run natively on a phone. Because only about 3B parameters activate per token, the models converse like large models but recall facts like small ones.

The engineering leans on a purpose-built .qpack container that packs experts at fixed stride so fetching one is a single pread, a bounded hot-expert cache using LFU-plus-recency eviction, and runtime-compiled Metal shaders so no toolchain is needed at build time and the same code ships on iOS. Roughly three-quarters of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, avoiding a growing KV cache at long context. Every stage of the forward pass is validated against mlx-lm reference implementations, and the project credits the earlier TurboFieldfare work for proving the expert-streaming thesis on Gemma.

The honest caveat is speed: the iPhone build currently manages about 1 token/second, with the decode loop bound by dispatch overhead rather than I/O, so there’s clear room to optimize. Swiftlet ships as a Swift library, CLI, and OpenAI-compatible loopback server, and powers an open-source iOS app (Priv AI) that runs entirely on-device with no server involved. It’s Apache 2.0, was built in collaboration with Claude Code, and points toward a future where capable local inference — and the privacy that comes with keeping data on the device — is practical on consumer hardware.