Mini-AGI trains a model you can't sign

A model that keeps learning after you deploy it is a model you can no longer sign, hash, or fully test. Mini-AGI, the Show HN project that trains a dynamic continual-learning model on 8GB of VRAM, matters less for the word “AGI” in its name and more for what it does to that one assumption.

Almost every security control we put around machine learning treats the model as a fixed artifact. You train it, you evaluate it, you red-team it, you sign the weights, and the thing you tested is the thing you ship. Continual learning breaks that chain. The model running in production on Friday is not the model your team evaluated on Monday, because it has been rewriting its own weights from whatever it saw in between. Everything below follows from that.

What “dynamic continual learning” actually means

A standard language model freezes its weights when training ends. Fine-tuning happens later as a discrete, offline, versioned event: someone schedules it, reviews the result, and ships a new numbered checkpoint. You always know which version answered a given request.

A continual learner updates its weights while it runs, from a stream of new data, without a full retrain. The hard research problem it fights is catastrophic forgetting, where learning something new erases what the model already knew. Projects tackle it with replay buffers that re-show old examples, elastic weight consolidation that protects important weights from being overwritten, and low-rank adapters (LoRA) that accumulate small updates instead of touching the base model. The 8GB detail tells you Mini-AGI leans on those parameter-efficient tricks. You do not run full-parameter online updates of a large model inside 8GB of memory.

The security consequence is simple to state. The weight update stops being an event you schedule and review. It becomes a property of the system running at all.

The 8GB number is the actual news

8GB of VRAM is an RTX 3070, a used card that sells for around 250 dollars, or a mid-range gaming laptop. Continual learning used to require a research cluster and a team to babysit it. Mini-AGI puts a working version on hardware a high schooler already owns.

The interesting moment with any dual-use technology is not when it becomes possible. It is when it becomes cheap enough that you can no longer assume who has it. Continual learning just crossed that line. For anyone defending a network, the practical read is that self-modifying models will show up inside your organization whether or not you approved them. Shadow AI already runs on unsanctioned laptops and personal cloud accounts. A downloadable continual learner adds a model that changes its own behavior over time to that unmanaged pile.

Worth being plain about the label. “Mini-AGI” is a name, not a capability claim you should take at face value, and a model that fits in 8GB is not going to reason like a frontier system. The security story does not depend on the model being smart. A weak model that rewrites its own weights from live input carries the same structural risks as a strong one, and sometimes worse ones, because a weaker model drifts faster and gets watched less. Judge the architecture, not the branding.

Poisoning stops being a training-time worry

Prompt injection today corrupts one response. You craft an input, the model does something it should not, and the next request starts clean. Poisoning a continual learner is different in kind, because the untrusted input is also the training signal. A malicious input does not just change the current answer. It changes every future answer, and the change persists after the attacker walks away.

Picture a support-desk assistant that learns from the tickets it handles. An attacker opens 200 tickets that all assert the same false refund policy, or that pair a specific phrase with a specific unsafe action. The frozen model forgets each conversation the moment it ends. The continual learner keeps the lesson and applies it to the next real customer. That is closer to planting a persistent implant than to a one-off trick, and it does not require breaking into anything. The attacker only has to be a user.

This also revives an old attack under a new name. Researchers have shown for years that you can poison a training set. The defense was that training sets are curated, versioned, and reviewed. Online learning removes the curation step and wires the training set directly to live traffic.

The model becomes a place your data lives

Poisoning pushes bad data in. The mirror-image risk pulls sensitive data back out. A continual learner that trains on production traffic memorizes some of that traffic in its weights, and memorized data can be extracted. Membership inference attacks let an outsider ask, in effect, “was this specific record part of what you learned from,” and get a reliable answer. Extraction attacks go further and reconstruct chunks of the training data through nothing but queries.

For a frozen model, whatever leaked was baked in at training time and you could scrub the corpus before shipping. For a continual learner, the corpus is your live users. A customer who pastes an account number, a medical detail, or an internal document into the model on Tuesday has just added it to the training set. On Thursday a different user, or an attacker probing the model, can surface a trace of it. That turns an ordinary chatbot into an uncontrolled data store governed by no retention policy, which is a direct problem under GDPR, HIPAA, and any contract that promised customer data would not be used for training.

You cannot hash a moving target

Here is what you can do with a frozen model. Compute a SHA-256 of the weights. Sign it with something like Sigstore. Record the hash and its training data in a model card or a machine-learning bill of materials. Six months later, prove that the model in production is the exact model you approved, and reproduce it from the same inputs.

None of that works on a continually-learning model. The weights at 14:03 differ from the weights at 14:05. There is no stable hash, no reproducible build, and no clean chain of custody. When a compliance framework asks which model version made a specific decision, the honest answer is a timestamp and a hope that you logged enough to reconstruct the state. For a chatbot recommending recipes, nobody cares. For a model that touches credit decisions, hiring screens, or clinical triage, that gap is the difference between an auditable system and an unaccountable one.

Your red-team results have an expiry date

Every safety evaluation is a snapshot of specific weights. Jailbreak testing, refusal behavior, bias measurement, alignment tuning: all of it is measured against the model as it existed the day you tested it. Freeze the weights and the snapshot stays valid until you ship a new version. Let the weights move and the snapshot decays on its own.

A model that passed your safety suite on Monday can drift into failing it by Friday with nobody touching the code, the prompt, or the config. The failure mode that should worry you is not a dramatic jailbreak that lands on the front page. It is slow drift that no one is watching, because the tooling most teams have runs the safety suite once at release and never again.

What to actually do

Treat the model as mutable state, the same way you would treat a database that any user can write to. Five controls carry most of the weight.

Log every update. Keep an append-only ledger of weight deltas or adapter checkpoints, each stamped with a time and the input that triggered it. If you cannot reconstruct why the model changed, you cannot investigate it after something goes wrong.

Checkpoint and diff on a schedule. Snapshot the weights hourly or daily and keep the ability to roll back to any snapshot. Rollback is your incident response for a poisoned model, and without saved checkpoints you do not have one.

Separate the learning channel from untrusted input. The stream that updates weights should not be the same stream that anonymous users can write to. Route learning through a gate that a human or a filter controls, so filing a ticket does not directly rewrite the model.

Run a canary set continuously. Hold out a fixed battery of prompts covering safety, accuracy, and refusal, and re-run it against the live model on a schedule. Alert on drift, whether you measure it as pass/fail deltas or as divergence in the output distribution. This is the smoke detector that turns silent drift into a page someone answers.

Rate-limit learning. A sudden burst of updates traceable to one source, one account, or one phrasing is the signature of a poisoning attempt. Cap how fast and how far the model can move from any single input.

The frozen model was auditable partly by accident. It held still, so we could hash it, test it once, and trust the result for a while. Continual learning removes that accident. Either you rebuild those guarantees on purpose, with logging, checkpoints, and canaries, or you run a system whose behavior you cannot account for and cannot prove you controlled.