Fitting an Autoregressive Language Model into 25KB on a 1975 8-Bit CPU

A hobbyist project squeezes a working next-token language model onto a MOS 6502, the 8-bit chip behind the Apple II and BBC Micro. The entire system lives in 25KB of user memory — roughly 9KB of inference code and 13KB of weights — and runs on the author’s father’s 1980s BBC Model B. Because the 6502 has no multiply instruction, the design leans on BitNet-style ternary weights constrained to {-1, 0, 1}, which turns each dot product into cheap add/subtract operations: a ternary accumulate costs about 30 clock cycles versus 150 for a full 8-bit multiply. Packing four 1.58-bit parameters per byte (chosen over five for fast bit-shift unpacking) yields about 52,000 parameters in 13KB, and the team found more low-precision weights beat fewer high-precision ones. Only the output projection head is kept at int4 for cleaner probability spread.

The architectural choices are driven by the 32KB RAM ceiling. Transformer attention is rejected because its KV cache grows with every generated token, so the model uses a fixed-size recurrent state instead. GRUs proved unstable — in the ternary regime the forward matrix’s spectral radius routinely exceeds 1, causing gradients to explode unless ~98% of weights are zero — so the author settled on Mamba, whose per-channel scalar decay is bounded below 1 by construction and cannot blow up. Activations stay in 8-bit with 16-bit accumulation to avoid overflow, and training uses a straight-through estimator that quantizes to ternary on the forward pass while keeping full-precision gradients on the backward pass.

The toolchain is a nice piece of retro engineering: models are trained on a MacBook, compiled from C to 6502 via CC65, verified for parity against the Python reference using the sim65 emulator, and loaded onto the real hardware through a DIY 3.5mm-to-tape cable that fools the BBC into thinking it’s reading a cassette. There’s also an in-browser demo built on the jsbeeb emulator that boots the machine and auto-runs the model, though generation takes a few minutes. The result is less about practical NLP than a demonstration of how far aggressive quantization and the right recurrent architecture can stretch severely constrained hardware.