Same commit, two builds, checksums that disagree

A build system checks out a commit, executes the instructions inside it, and produces an artifact. The artifact is signed. The signature is recorded against the commit hash. The deployment system accepts the artifact because the record points to a commit, and the commit is immutable. Every link in that chain behaves correctly. Every record is accurate. The pipeline did exactly what it was configured to do.

Run the same build against the same commit twice, separated by a week, and the two artifacts do not match. Their checksums differ. Nothing in the repository changed between the two runs. The system records both artifacts as products of the same commit, and both records are, by the system’s own definition, correct. The provenance chain is intact in both cases. It points to the same place.

Inspect the deployed artifact and most of its contents appear in no commit in the repository. Dependency trees resolved at build time, base layers pulled at build time, toolchain output generated at build time. The repository contains a few thousand lines that name these things. The artifact contains millions of lines that are these things. The system stores the hash of the former and presents it as the identity of the latter. This is not a malfunction. It is the recorded, observable, everyday behaviour of the pipeline.

The assumption was that a commit is a complete description of the software. Version control is built on content addressing. The hash is derived from the content, so an identical hash guarantees identical content. Within the boundary of the repository, that guarantee is mathematical. It does not degrade, it does not drift, and it cannot be forged without breaking the hash function itself. The system extended its trust model outward from this one hard guarantee.

The trust model assumed persistence. Whatever was true of a commit at the moment it was evaluated would remain true at every future moment, because the commit itself could not change. A review performed once held forever. A scan performed once held forever. The immutability of the object was treated as immutability of every conclusion ever drawn from the object. Trust was resolved a single time, at the commit, and then carried forward indefinitely.

The trust model also assumed transferability. Everything reachable from a trusted commit inherited the commit’s trust. The build instructions lived in the commit, so the build was trusted. The dependency manifest lived in the commit, so the dependencies were trusted. The base image was named in the commit, so the image was trusted. The boundary of the hash was assumed to be the boundary of the software. Inside that assumption, pinning a reference and possessing the content were the same act.

It did not start as a fiction. Early on, the repository and the software were nearly the same object. The commit described most of what ran, and the gap between reference and content was small enough to ignore. What changed was not the integrity of commits. Hashes still hold. What changed was the validity of the assumption that the commit contains the software it names.

Over time, the proportion of the running artifact actually described by repository content collapsed. The manifest names versions; a registry supplies the content. The lockfile pins package hashes; the build still pulls compilers, plugins, and base images resolved through tags that move. The commit became a set of assembly instructions executed against external state, and that state is read at build time, not at commit time. The software is made between commits, in the interval between when a reference is written and when it is resolved. Two resolutions of the same reference are two different events, and the system treats them as one.

The system never re-evaluated this. Trust still resolves at the commit, exactly as designed. Each build inherits conclusions drawn from an evaluation performed against content that no longer exists in the form it was evaluated, and applies those conclusions to content that did not exist when the evaluation occurred. The assumption no longer holds. The system has no way to register that it no longer holds, because the one property it checks, the hash of the repository, remains perfectly true. The guarantee is intact. The thing it was assumed to guarantee has moved.

The mechanism is substitution. At every point where the pipeline could evaluate content, it instead checks a reference, and the check passes because references are cheap to satisfy. The build system does not evaluate the dependency it downloads. It evaluates whether the name and version of the downloaded object match the name and version in the manifest. The deployment system does not evaluate the artifact. It evaluates whether the signature on the artifact corresponds to a commit hash in the record. Each check is real. Each check is performed. Each check confirms identity of source rather than integrity of content. The system optimized for the property it could compute cheaply, and the cheap property became the only property it knows.

The signature attests to exactly one fact: that a particular build system, at a particular moment, produced this artifact while pointed at this commit. It does not attest to what the build system read while producing it. Content that arrives through resolution at build time enters the artifact carrying the trust of the commit that named it, and exits the pipeline under a signature that names the commit again. The reference appears at both ends of the chain. The content appears at neither. In practice, the provenance record is a record of which instructions were followed, presented as a record of what was built.

Nothing is bypassed when this goes wrong. Content substituted behind a resolved reference does not defeat the pipeline. The pipeline retrieves the substituted content, incorporates it, signs the result, and records the provenance, exactly as configured. The resulting artifact carries a valid signature and a true provenance record. Every control fires. Every control passes. The compromise travels through the system as expected behaviour, because from the system’s perspective it is expected behaviour. The system was asked to confirm that the artifact came from the commit, and it did. Whether the artifact contains what the commit was once believed to describe is not a question the system was built to ask.

The failure is invisible to the controls surrounding it for a structural reason. Artifacts already differ between builds as a matter of routine. Two clean builds of the same commit produce different checksums, so a changed checksum carries no signal. A substituted dependency produces a changed checksum, which is what every build produces. The audit log shows a successful build. The provenance chain is unbroken. The signal of compromise is bitwise indistinguishable from the signal of normal operation, because normal operation already includes content the system never examined. The ambient non-determinism of the pipeline is the cover under which substitution moves.

The pattern is execution based on reference, not verification. A system performs an expensive evaluation once, binds the result to a stable identifier, and thereafter executes against the identifier. Resolution of the identifier into content happens later, repeatedly, against external state, and each resolution inherits the original evaluation without repeating it. The identifier is stable. The content is not. The system’s confidence is a property of the identifier, so the confidence is stable too, regardless of what the content has become. The evaluation and the execution drift apart in time, and the identifier bridges the gap by asserting that no gap exists.

The same mechanism operates in pharmaceutical manufacturing. A drug is approved against a specification: a document naming ingredients, sources, and acceptance tests, evaluated once and in depth at the moment of approval. Production then runs for years against that document, resolving each named ingredient through external suppliers at manufacture time. Incoming material is checked against the identity tests written into the specification, tests that confirm the material matches the reference signature, not that the material is the substance the original evaluation examined. In 2008, an adulterant entered the heparin supply chain that passed those identity tests. Batches were produced, released, and administered with every control passing, because every control checked correspondence to the reference. The approval was an evaluation of content that existed once. Production resolved the reference against content that had changed. The system had an operation for resolving. It had no operation for re-evaluating.

In both cases the reference is honest throughout. The commit hash is mathematically true. The specification is exactly what was approved. The corruption enters at resolution, in the interval between writing the reference and resolving it, and the records remain accurate the entire time because the records describe the reference, not the content. The pattern requires no defective component. It requires only three conditions: time, an external resolution step, and a trust model that treats one evaluation as permanent. Wherever those three coexist, the artifact is made between commits, in the gap the identifier insists is not there.

The hash guarantee never broke. It was never extended. The system holds one cryptographic truth about a few thousand lines of instructions and presents it as assurance over millions of lines of resolved content. The distance between those two numbers is the exposure, and it is recorded nowhere, because the system measures the reference and the distance exists only in the content.

The verification did not disappear. It moved. It moved backward in time, to the moment the reference was written, while the software moved forward, to the moment the reference is resolved. Every build widens the interval between the two.

The system resolves trust once, at the commit. It does not revalidate at resolution. The control exists. The outcome does not.