Mini-AGI trains a model with no frozen base

Updated 11 October 2026: the 19% / 0.634-nat figures were the live status as of 8 October, not the 1 October snapshot, which stood at about 11%.

mini-AGI takes a gradient step on every chunk of text it reads, through the same code path it uses to answer you. There is no frozen base and no separate fine-tuning stage; reading and being trained are the same forward pass. Posted to Show HN by its author, volotat, it is a byte-level continual-learning language model that trains from scratch on a single 8 GB consumer GPU and keeps learning from whatever stream you point it at. That design is the pitch, and it is also why the model has no single frozen version to sign: there are only snapshots of a process that keeps running.

Start with how it runs. The alphabet is the 256 byte values, so there is no tokenizer and no vocabulary to fit. Each character passes through two dense prelude blocks and then a single recurrent block applied up to 24 times, each application picking its own top-8 experts from the 32 the card holds. A halting head decides when another row would not change the answer, so easy characters take one pass and hard ones take many; the live run averages around twelve. Every expert is an ordinary file on disk holding its weights and its Adam moments, and only the experts a forward actually uses are paged onto the GPU. Parameter count is bounded by free disk space rather than VRAM. When the model runs short of capacity it grows new experts, and it prunes the ones nothing asks for.

The author is clear that this is toy-level. As of 8 October 2026 the live run was 19% through its first pass over a 7.9-billion-character corpus, with a held-out loss of 0.634 nats and a perplexity of 1.89; the undertrained checkpoint uploaded to Hugging Face on 1 October was taken at about 11% (0.668 nats, perplexity 1.95), and the README’s status line keeps moving. A generation sample from early in the run reads like a child who can spell: “the cherry tree saw another cherry. It was very big, and it had long, shiny wings.” The interesting part is the mechanism, not the capability.

Now the provenance problem. Everything in a model supply chain starts from a frozen artifact: you train, you stop, you hash the weights, you sign the hash, you distribute, and the consumer verifies the signature matches the bytes. mini-AGI is built not to stop. The weights move as it trains, so the live model is a moving target rather than one fixed set of bytes. The way you get a fixed artifact out of it is to snapshot, which is what the author has done: the undertrained copy posted on 1 October is a point-in-time snapshot, and the moment the running model reads more, that copy and the live model have diverged. Two people who start from the same weights and feed them different streams end up with different models, because the stream the model reads is part of what makes it.

It is not only the weights that move. The architecture is not fixed either: the model grows new experts when it runs short of capacity and prunes the ones nothing asks for, so the expert count tracks the data. Over the life of the running process both its size and how much it has read are open-ended by construction. A snapshot captures one size and one amount of reading; the process behind it holds neither fixed.

For anyone thinking about running one, the operational consequence is that the text you let it read is the training set, with no gate in between. The author frames continual learning from “everything it reads” and “every conversation you have with it” as the whole point. To an attacker that is a data-poisoning channel with no review step: while the model is training, whatever it reads is fed straight into the weights, and because reading and training are the same event there is no frozen base held alongside the live model to revert to. Going back means restoring a snapshot you took earlier.

The ownership claim the project leads with is the same fact seen from the other side. Because the model is files on your disk that learn from your data, nobody else can take it away or switch it off. The flip side is that the thing you own is a running process, not a fixed artifact; what you would hand to someone else is a snapshot of it, the way the author posted the undertrained copy.

The one boundary the design does hold is that generation is read-only on the weights. The model does not train on its own output and takes no gradient step while writing. The author verifies this at the plumbing level: an expert leaving the GPU is copied back only if the optimiser actually stepped it, and a reply loads experts at nearly every character while stepping none, so writing persists nothing to disk. Inference does not drift the model; only reading does. If you want a fixed version to point other tools at, you snapshot the directory, as the author did with the undertrained weights. The “final” weights are not out yet: the author says the run is still on its first pass over the corpus, about a fifth of the way through as of 8 October and several weeks from done at the current rate, with the published weights going up once it has read all of it.