28.9M-parameter LLM runs on an $8 ESP32 by keeping most of itself in flash

A developer has fit a 28.9-million-parameter language model onto an ESP32-S3, an $8 microcontroller, and gets it generating text at about 9 tokens per second entirely on-device — no server involved. That’s roughly 100x larger than the previous 260K-parameter model people had squeezed onto comparable hardware. The trick is memory placement. A microcontroller has almost no fast RAM (512KB of SRAM here), and conventionally the whole model must be reachable from it, which caps model size. Instead, the bulk of the parameters — a 25-million-row embedding table the model only reads from — stays in slow but plentiful flash, and each token pulls just the handful of rows it needs (about 450 bytes). The small compute-heavy core that does the actual reasoning stays in fast memory.

The approach borrows Per-Layer Embeddings from Google’s Gemma 3n and Gemma models, applied here to a microcontroller’s memory hierarchy rather than a phone or GPU — apparently a first on a chip this small. SRAM holds the thinking core used every token, PSRAM holds the output head and scratch space, and flash holds the giant table sampled a few rows at a time, so most of the model costs almost nothing to run because it’s never loaded.

The point is the architecture, not the model’s abilities. Trained on TinyStories, it writes short, mostly-coherent children’s-story text but can’t answer questions, follow instructions, code, or recall facts — a limit set by the tiny reasoning core, which the memory trick doesn’t touch. The project grows out of Karpathy’s llama2.c lineage, and the author deliberately kept the messy commit history, including a parameter-counting bug that had inflated an early figure and the corrected number that followed.