TurboFieldfare streams a 26B MoE model on 8GB Macs in ~2GB of RAM

TurboFieldfare is an open-source Swift and Metal runtime that runs the instruction-tuned Gemma 4 26B-A4B mixture-of-experts model on Apple Silicon without loading its full 14.3GB weights into memory. It keeps the shared 1.35GB core and an FP16 KV cache resident, then pulls only the experts each token actually routes to off SSD on demand. That expert-streaming trick drops the working-set footprint to roughly 2GB, low enough to run on an entry-level 8GB M2 MacBook Air — the developer’s validated target — rather than the high-memory machines large local models usually demand.

Unlike most local-inference projects, the runtime isn’t a wrapper over MLX or llama.cpp; it’s hand-built and model-specific, with Metal kernels handling attention and the router while a 16-slot LFU cache and bounded parallel disk reads fetch missing experts. A streaming installer pulls byte ranges directly from a pinned Hugging Face revision and repacks them into the runtime’s .gturbo format, avoiding a second full copy on disk. The project ships a native Mac app, a CLI, and an experimental loopback OpenAI-compatible server that supports chat completions, streaming, and client-authorized tool calls. It’s text-only, arm64-only, and requires the newest toolchain (macOS 26, Metal 4, Swift 6.2).

The significance is less about one model than the engineering pattern: for sparse MoE architectures where only a fraction of parameters activate per token, aggressive SSD streaming can decouple model size from RAM requirements. That reframes what hardware can run a 26B-class model locally, trading memory pressure for I/O and making capable on-device inference viable on commodity, memory-constrained laptops. The author frames the published benchmarks as reference points rather than performance ceilings and invites community results across other Apple Silicon machines.