Swift binary streams a 104GB Qwen MoE off SSD to run it on a 48GB Mac at ~12 tok/s

slotstream is a single Swift binary (no Python) that runs Qwen3.8-Flash-Next — a 125B-parameter mixture-of-experts model weighing 104GB on disk at 4-bit — on Apple Silicon Macs that can’t fit it in memory. It does this by streaming the routed experts from SSD into a fixed pool of cache slots shared across all 48 layers, sizing itself to whatever RAM you give it. On the one machine it’s actually been measured on, a 48GB M5 Pro, warm decode runs about 12 tok/s. It exposes the Ollama and OpenAI chat APIs on port 11434, so existing clients like Open WebUI and the OpenAI SDKs work unchanged; unsupported features (tools, images, JSON-schema output, logprobs) return an explicit 400 rather than failing silently.

The core engineering insight is why naive memory-mapping fails: Apple’s MLX framework can’t materialize part of an mmap’d tensor, so a top-10 expert gather forces evaluation of all 512 experts in a layer, dragging ~100GB into memory and pushing the machine into swap without ever producing a token. slotstream sidesteps this by reading experts with pread into a bounded cache, keeping only the 3.8GB dense trunk resident. Cache size affects speed but never output — greedy decoding is byte-identical across cache sizes, and the engine resizes the cache under memory pressure between requests while keeping output stable. Practical friction is dominated by disk and bandwidth: you need roughly 110GB free (a 512GB Mac is the realistic floor), downloads run over eight resumable, sha256-verified TCP connections, and prompt prefill is the slow axis — 8,000 tokens take about a minute on the 48GB machine, with a prefix cache making follow-up turns cheap.

The project leans hard on reproducibility and honesty about its limits: releases ship with signed build provenance you can verify via gh attestation, weight hashes are compiled into the binary, and a doctor command prints your machine’s plan before anything downloads. An optional multi-token-prediction draft head offers 1.5–1.9x speculative decoding on machines with spare memory. The stated status is candid — v0 runs exactly one model, one process at a time, and every performance tier smaller than the 48GB anchor is an extrapolation from its curve, not a real measurement.