How a decade-dead Xeon was patched to run Gemma 4 26B at reading speed, no GPU

Ryan Findley got Google’s 26-billion-parameter Gemma 4 mixture-of-experts model generating text at roughly five tokens per second on a thirteen-year-old HP StoreVirtual box — two Ivy Bridge Xeons, 128GB of slow DDR3, and no GPU. The starting point was ikawrakow’s ik_llama.cpp fork, which packs the speculative decoding, CPU-side flash attention, and MoE routing tricks that make CPU-only inference viable. It refused to run: the fork’s fast kernels assume AVX2 and FMA3, instructions that didn’t ship until Intel’s 2014 Haswell generation, and the box’s older ‘v2’ chips only have AVX1.

The interesting part is the debugging. With GGML_USE_IQK_MULMAT disabled, most of the codebase fell back to scalar math, but two fused gate+up MoE operations were still emitted by the graph builder while having no matching case in the compute dispatcher. Every forward pass therefore left roughly 240 expert-FFN tensors reading uninitialized memory, and the model spat out fluent-looking multilingual gibberish. What cracked it was determinism: instrumenting the raw logits showed a suspiciously clean positive bias — a mean logit near +16 and 80% of the vocabulary skewed positive — the signature of uninitialized floats rather than random corruption. The fix reworks the graph builder so that when the fast path is off it splits the combined weight tensor and emits stock ggml ops (mul_mat_id plus a fused SILU-multiply) that actually have scalar implementations, guarded so AVX2 builds stay bit-identical.

Findley is candid that he doesn’t write C++; the diagnosis and patch came from a Claude instance running on the server itself, while his role was running experiments and recognizing when the output was correct. The three-commit patch is open as ik_llama.cpp PR #2138, awaiting maintainer review. Beyond the hardware stunt, the piece argues that being ‘good with AI’ should mean aiming a model at an unpackaged problem and judging whether the answer holds up — and it’s a useful signal that old enterprise iron can serve as a free local fallback when paid APIs are down or per-token batch jobs don’t pencil out.