The green check mark proves nothing

Opening Claim

Working code is not vetted code. Those are two separate states, and treating the first as proof of the second is the decision that gets you breached. I reject AI-generated code that passes its tests, because function is not the bar. The bar is whether I can account for every line it produced, every dependency it pulled in, and every assumption it encoded into the execution path. “It works” tells me one thing only: the happy path executed once, under conditions I controlled. It tells me nothing about what the code does when those conditions change, what it trusts, or what it reaches for when a similar prompt has appeared ten thousand times in a corpus I never audited.

Acceptance is an authorization decision. When you merge code, you extend your trust boundary to include whatever produced it. The code then inherits your identity, your access, and your execution context. It runs as you. So the question is never “does it work.” The question is “what am I authorizing to run inside my boundary, and can I prove what it does.” If you cannot answer the second part, the first part is irrelevant. A correct output you cannot account for is still an unaccounted execution path running under your privileges.

This is a posture statement, not a verdict on AI quality. The model can be good. The output can be correct. The risk does not live in the tool. The risk lives in the act of accepting output from a source you did not vet, on the strength of a signal that was never built to detect the thing that hurts you. “It works” is a behavioral signal. Trust is a provenance question. Using one to answer the other is the gap. Everything that follows comes from that gap.

The Original Assumption

The assumption running underneath most engineering pipelines is that passing behavior implies safe behavior. Code that compiles, runs, and returns the expected output is treated as code that has been reviewed. The test suite becomes the proxy for trust. The green checkmark appears, the change is considered accountable, and it moves forward. That chain holds under one specific condition: a human authored the code with intent you can interrogate. Remove that condition and the chain has nothing holding it up.

Tests validate the paths you wrote tests for. They confirm that defined inputs produce defined outputs. They do not confirm the absence of an unintended path. A function that returns the correct value can also weaken a comparison, widen a permission, log a secret, open a connection, or pull a dependency you never named. None of that breaks the test, because the test was never watching for it. “It works” is a statement about the path you observed. It is silent on every path you did not. Passing is the absence of a specific failure you predicted, not the presence of safety.

When a human writes the code, you hold a second control that rarely gets named: authorship. You can ask why a line exists. You can trace intent. The author is accountable, and the reasoning is recoverable. That second control is what the original assumption quietly depends on. The checkmark was never the real boundary. The accountable author behind it was the boundary, and the checkmark was just the visible marker on top of it. Strip out the author and the marker is now carrying enforcement weight it was never designed to carry. That is the load-bearing failure, and it is invisible until you change who is writing the code.

What Changed

The author is now an unvetted source. AI-generated code arrives with no interrogable intent. There is no person who decided why a specific construct was chosen, no reasoning you can recover, no accountability you can trace back to a decision. The output is shaped by a training corpus you did not audit and cannot inspect. You receive the artifact stripped of the exact context that made authorship a control. So the second control is gone, and the first control, “it works,” was never sufficient on its own. You are now gating trust with a signal that only ever measured behavior, against a source whose provenance is unconfirmed by definition.

Scale changes the exposure. A human writes code at human speed, and that speed bounds how much unreviewed material can enter a system per day. Automated generation removes the bound. The same mechanism that produces correct code at volume produces weak or wrong code at the same volume, through the same trusted pipeline, under the same identity, gated by the same checkmark. Automation scales control and failure on the same curve. If the acceptance gate is “it works,” then everything that works and is wrong clears that gate at machine speed, and it clears it wearing your authorization.

The surface this opens is not hypothetical, and it does not require a hostile model. A source that draws on public training data inherits whatever was common in that data, including insecure patterns that recur because they are easy to write, not because they are safe to run. Dependency names can be produced that do not exist in any registry, and a name that does not exist is a namespace someone else can claim. Constructs can be reproduced that were correct in their original context and unsafe in yours, because context does not travel with the snippet. You do not need malice in the tool for any of this to land. You need three conditions present at the same time: an unvetted source, an acceptance signal that does not test for trust, and an execution context that runs the output as you. All three are present right now. If the system allows it, it will happen.

Mechanism of Failure

The failure is not a single bad merge. It is a migration of the acceptance threshold, and it happens without anyone deciding to move it. The signal “it works” is cheap to produce and cheap to read. Verifying authorship is expensive, because it requires understanding why each construct exists. When the source generates correct output at volume, that cost asymmetry pulls review toward the cheap signal. The team’s working definition of “reviewed” moves from “I can account for this” to “the checkmark is green.” That migration is the mechanism. It is invisible because every individual merge looks identical to the one before it. The marker on top is unchanged. The accountable author behind the marker is gone, and nothing in the gate reports the difference.

The drift compounds because accepted code becomes context for the next decision. Code already in the repository is treated as reference. A construct that entered without provenance gets read as precedent, copied, extended, and depended on. The unaccounted path is now load-bearing. Removing it later requires exactly the accountability you skipped at merge, except now there is more built on top of it. The cost of verification does not disappear when you defer it. It moves downstream and increases. You pay it during incident response instead of during review, under worse conditions, with less context, after the path has already executed under your identity. Deferred verification is not avoided verification. It is verification relocated to the most expensive possible moment.

Name the enforcement point that broke. The control that failed is authorship verification, and it failed at the acceptance gate. The gate was enforcing behavior. Behavior was never the boundary. A control that checks the wrong property is not a weak control. For the property that matters, it is an absent one. The green checkmark did not degrade and was not bypassed. It was never measuring provenance in the first place. The failure is structural: a behavioral gate is being used to make a trust decision, repeatedly, at machine speed, with no second control standing behind it. The marker is now carrying enforcement weight it was never built to hold, and it holds none.

The Same Mechanism, Other Sources

Reduce the failure to its conditions and it stops being about AI. Three properties are present: an unvetted source, an acceptance signal that measures behavior instead of provenance, and an execution context that runs the accepted output under your identity. Wherever those three hold, the same exposure holds. AI-generated code is one instance. The third-party dependency is another. A package is accepted because it installs and the build stays green. The source is a registry artifact whose provenance you did not verify. On install and at runtime it executes under your identity and reaches your access. The acceptance signal, “the build passed,” measures behavior. It says nothing about who authored the package or what its install step does. This is the identical mechanism wearing a different label.

This is the same mechanism, not a similar concept. The shared property is precise: a behavioral pass-signal standing in for a provenance check, with acceptance granting execution context. The dependency name that does not exist in any registry, named earlier, is this mechanism at its sharpest. The generator emits a plausible name. The name is unclaimed. Accepting the name as real authorizes whatever later occupies that namespace. Trust was extended to a string before the string pointed at anything. Provenance was never checked, because the gate only checks behavior, and an unresolved name has no behavior to fail yet. The gate cannot fail a thing that has not run. So it passes, and the authorization is granted in advance of the code that will use it.

Generalize the rule and the tool becomes interchangeable. The pattern is not “AI is risky” and not “dependencies are risky.” The pattern is that any acceptance gate reading a behavioral signal to make a provenance decision will pass unvetted execution into your boundary at the rate the source produces it. Change the source from a model to a registry to a pasted snippet and the three conditions still hold, and the outcome still holds. If the gate measures behavior and the decision requires provenance, the gap does not depend on which source you are looking at. It depends on the gate. Fixing one source while leaving the gate intact moves the exposure, it does not close it.

Operator Position

Acceptance is an authorization decision, so it must be gated on accountability, not behavior. The merge bar is fixed: I can account for every line, every dependency, and every assumption encoded into the execution path. If I cannot account for it, it does not enter the boundary. Function is a precondition, not the gate. “It works” qualifies code to be reviewed. It does not qualify code to be merged. Those are two separate states, and the entire failure described here comes from collapsing them into one. I keep them separate on purpose, because the separation is the control.

Identity is the boundary, and trust across it must be continuously validated. The code runs as you, so the only question at the gate is what you are authorizing to run under your privileges and whether you can prove what it does. Provenance is the control that answers it. When the source cannot be vetted, the output must be vetted line by line before it inherits your identity. There is no configuration of an unvetted source plus a behavioral gate that produces an accountable system. Controls that are not enforced are not controls. A review that confirms the checkmark is green is not enforcing the boundary. It is decorating it, and decoration does not stop execution.

If the system allows unaccounted code to merge on a behavioral signal, unaccounted code will merge. At volume. Under your identity. The model being good does not change this. The output being correct does not change this. Both are true and both are irrelevant to the decision, because the decision is about provenance and they are statements about behavior. I reject AI-generated code I cannot account for, even when it works, because the thing being rejected is not the code. It is the unverified authorization attached to it. Account for it, or do not run it. There is no third state, and treating “it works” as if it were the third state is the gap everything in this briefing came out of.