Mini-AGI: a byte-level model that trains from scratch and never stops learning on 8GB VRAM

Mini-AGI is an experimental continual-learning language model designed to be trained end-to-end on a single consumer GPU with as little as 8GB of VRAM. Its central trick is treating weights as ordinary files on disk and paging only the active ‘working set’ onto the card, so total capacity is bounded by free disk space rather than VRAM. Rather than a fixed stack of layers, each byte passes through two dense prelude blocks and then a recurrent block applied up to 24 times, with a PonderNet-style halting head deciding how much depth each character needs and a soft top-8 mixture-of-experts pool that routes per block-application. The architecture grows new capacity when it runs short and prunes experts nothing asks for, and reading and training are the same forward pass — there is no separate fine-tuning stage or frozen base.

The design is a direct response to a limitation of today’s owned models: anything you can run locally is something someone else trained and froze, and continued training on your own data tends to trigger catastrophic forgetting. Mini-AGI’s pitch is a model that trains on your hardware, keeps learning from every interaction, and can’t be switched off by a vendor. Working at the raw byte level (256 possible values) sidesteps tokenizers and vocabulary limits entirely, and rotary positions with no learned parameters let the context window grow through continued training. Engineering details reinforce the continual-learning goal — each expert carries its own Adam optimizer moments so momentum isn’t handed to a stranger during swaps, and experts stay in their VRAM slot by identity to minimize reloads.

The author is candid that this is a toy-scale experiment, not a frontier system. At roughly 243M characters of training it produces grammatically correct, on-topic text that still loops and repeats, and no weights have been published yet — the first full pass over the corpus is still weeks from finishing. The significance is less about capability than about proof of concept: continual learning from a single data stream without catastrophic forgetting, on hardware almost anyone owns, pointing toward genuinely personal models whose ceiling is set by your data, scale, and patience rather than a fixed release.