Colibrì: a 1,300-line C engine runs a 744B model on a 25GB PC by streaming from disk

Colibrì is a dependency-free inference engine, written as a single ~1,300-line C file, that runs GLM-5.2 — a 744-billion-parameter mixture-of-experts model — on ordinary consumer hardware with roughly 25 GB of RAM. It exploits the sparsity of MoE models: only about 40B parameters activate per token, and only the ~11 GB of routed experts actually change token to token. The dense components (attention, shared experts, embeddings) stay resident in RAM at int4 quantization, while the ~370 GB of routed experts live on disk and are streamed on demand through an LRU cache, an optional pinned hot-store, and the OS page cache. No BLAS, no Python at runtime, no GPU — just AVX2 integer kernels on the CPU.

The implementation is unusually complete for a hobby project. It reproduces GLM-5.2’s architecture faithfully — MLA attention with a compressed KV cache 57× smaller than a naive layout, a DeepSeek-V3-style sigmoid router, weight absorption for decode, and the model’s native multi-token-prediction head for speculative decoding, which the author reports reaches 39–59% draft acceptance at int8. An offline converter turns the 756 GB FP8 checkpoint into the int4 container one 5 GB shard at a time, so the full model never has to exist on disk at once, and a pre-converted build is posted on Hugging Face. A ‘learning cache’ records which experts a user’s prompts actually route to and auto-pins the hottest ones, so the engine measurably speeds up with use.

The honest caveat is speed: decode is disk-bound, with a cold token costing roughly 11 GB of random reads, so this is emphatically not fast. The point is feasibility — a frontier-class model answering correctly on a machine that costs less than an H100’s cooling fan. The work is a striking demonstration that aggressive quantization plus disk streaming can democratize access to models otherwise locked to datacenter GPUs, and the author is explicitly soliciting benchmark numbers from people running it on better hardware.