Rewriting weights live on 8GB of VRAM

mini-AGI stores its weights as ordinary files on disk and pages only the experts a given forward pass touches onto the GPU, so its parameter count is bounded by free disk space rather than by VRAM. It targets a PC or laptop with at least an 8GB card, trains from scratch on that card, and does not stop training. It reads a stream of bytes one chunk at a time, takes a gradient step on each, and serves generation through the same code path. There is no frozen base and no separate fine-tuning run: reading and being trained are the same event.

The author, who posts as volotat, is blunt that this is a toy. The name is a half-joke about traits rather than a claim about capability. The point of the experiment is narrower and more interesting: continual learning from a single stream of data, without catastrophic forgetting, on modest hardware.

The alphabet is the 256 byte values plus nine markers for structure. There is no tokenizer to fit and no vocabulary that a new data type would break, so the model can read anything you point it at. Three constraints drive the rest of the design. It has to fit on 8GB without quantisation, because training needs gradients and optimiser state, roughly three times the weights again; that is why the weights sit on disk and only the experts in use occupy the card. It has to avoid catastrophic forgetting, because it learns from everything it reads. And it has to read arbitrary bytes, which is why it works at the byte level.

The architecture assembles itself

A character does not pass through a fixed stack of layers. It goes through two dense prelude blocks and then one recurrent block applied up to 24 times, each application choosing its own top-8 experts from a shared pool. At the current step that pool holds 397 experts; the card has 32 slots. A halting head scores each character at each row and stops it as soon as another row would not change the answer, so easy characters take one row and hard ones take many. Training uses the same mechanism through the PonderNet recipe, weighting each row a character runs by its halting probability so the halting head learns through those weights.

Routing happens per block-application, not per character. Each of the up-to-24 recurrent applications a character runs picks its own eight experts, so a single character touches far more of the pool than “top-8” suggests, and the same expert can be chosen several times at different depths. No expert is assigned a subject. Soft top-k routing spreads capability across the pool on its own, which means capabilities share parameters and can interfere.

How the paging actually works

Selection is one rule: the text chooses. Every forward’s first pass adds its characters’ requests to a running vote over everything read since position zero, and the 32 slots fill with the experts that text has voted for most. Each character then routes with the same router over the admitted 32 and takes its top eight of those. An expert on disk is scored exactly like one in VRAM, because the router rows for every expert stay on the card.

Each expert is a file holding its weights and its Adam moments. The moments belong to the expert, not to the slot it happens to occupy. Stepping an expert with whatever its slot last held would hand it the previous expert’s momentum, and training would keep looking healthy while every swapped expert inherited a stranger’s history. So moments travel only to a step: just before each optimiser step, every expert on the card that is not already holding its own moments loads them from RAM or disk. An expert already on the card stays in its slot; a newcomer takes an empty slot first, then the one admitted longest ago. Consecutive windows of one subject ask for nearly the same experts, so over fifteen minutes of real training a step loaded 2.4 experts on average. An expert nothing trained is never written back, which matters for generation: writing loads experts at almost every character and steps none of them, so it moves no moments and writes nothing to disk.

Left alone the router keeps choosing the same experts; reading held-out text on its own it put just 59 of 118 experts on a card across every subject. The loss carries a balance term, the Switch Transformer’s with one change. Switch charges each expert by its share of the current batch; here the share is taken over the last thousand or so training forwards, because a training window is a single text and a text should be free to want few experts. What has to be even is use across texts. At weight 0.01 the experts the router used across held-out text rose from 59 to 114, at a cost of 0.023 in held-out loss over those fifteen minutes.

Where the run is now

The published run has read 1,869.0M of 7,880M characters, about 24% of a first pass, at a held-out loss of 0.6188 nats and a perplexity of 1.86. The samples show the shape of a model this early. The chat prompt “What are you?” gets a coherent self-description. Greedy decoding with no guard collapses into repetition on code and arithmetic (a merge_sorted prompt repeats “sorted array of” for a full line), and the repetition guard that the sample log calls the adapted reading cleans most of that up, turning the arithmetic into runs of correctly formatted sums. The story continuation spells every word and stays on topic while producing a cherry that has “long, shiny wings,” which is a fair picture of the model at the point the clip was recorded.

The final weights are not published. The run is still on its first pass over the corpus, several weeks out at the current rate, and the weights only go up once it has been through all of it. If you want to run it as it stands, the most recent undertrained snapshot (12 Oct 2026) is on Hugging Face as Volotat/mini-AGI-undertrained, and because training is the same path as inference, you can point your own copy at a corpus and keep it going.