RC RANDOM CHAOS

The model kept no receipts

The "OpenAI stole my proof" fight is really a provenance gap in AI, and that gap is a safety problem, not just a credit dispute.

· 7 min read
The model kept no receipts

A large language model does not keep the papers it trained on. It keeps weights: a compressed statistical average of millions of documents. When it prints a proof, it cannot tell you which source it leaned on, because that link was never stored. That one fact sits under every “OpenAI stole my proof” story, including the one going around now, where researchers say a model reproduced a substantial result close enough to their own to ask where it came from.

The word “another” in the accusation matters more than any single case. This is not a one-off. It is a category of dispute that will recur every time a frontier model produces something a human recognizes. So it is worth separating the accusation from the mechanism, because most of the argument online is people talking past each other.

The word “stolen” is doing three jobs at once

When someone says a model stole a proof, they usually mean one of three different things, and the three carry very different weight.

  • Copying. The output matches a specific person’s text or argument line for line. This is the strongest version and the easiest to check. You put the two documents side by side and look.
  • Uncredited synthesis. The model recombines ideas from a known paper without naming it. The original author recognizes the shape of their work, but it is not a literal copy.
  • Convergence. Two parties reach the same result because the math left no other road. Given the same lemmas, competent mathematicians land in the same place. So do competent models trained on those mathematicians.

From outside the system, these three look identical. A lab publishes a proof, a mathematician says “that’s mine,” and there is no log that settles which of the three happened. That is the real problem. Not the accusation itself, but the absence of anything that could confirm or kill it. Every party ends up reasoning about a black box from its outputs, which is a bad position for the accuser, the accused, and anyone trying to referee.

OpenAI often cannot answer the question either

Here is the part the outrage cycle skips. Even acting in complete good faith, OpenAI frequently cannot prove where a given output came from. A transformer trained on a web-scale corpus has no index mapping outputs back to the documents that produced them. The methods that try to build that link after the fact, influence functions and training-data attribution, estimate which training examples most shaped an answer. They are expensive, approximate, and almost never run on production systems fielding millions of queries a day.

So when a dispute lands, both sides argue from inference. The researcher says the overlap is too specific to be chance. The lab says the model generalizes and nobody targeted that paper. Neither can produce a receipt, because the machine was built to generate text, not to remember its sources.

The same root cause produces benchmark contamination: a model trained on data that quietly includes the test set, then “solves” that test. There is no provenance boundary between what went in and what comes out, so nobody can cleanly say whether a result was earned or memorized. A proof dispute and a contaminated benchmark are the same gap showing up in two places.

Provenance is a safety problem, not only a credit problem

The copyright fight gets the headlines. The safety community should care more about the gap underneath it, because it is the same gap that shows up everywhere the stakes are higher.

If you cannot trace an output back to its inputs, you cannot audit the system. You cannot answer “why did the model produce this,” which is the question behind every incident review, every alignment claim, and every case of a model complying when it should have refused. An attribution failure and a safety failure are the same missing capability wearing different clothes. A system that cannot show its work on a math proof also cannot show its work when it recommends a drug combination, denies someone a loan, or emits code with a planted backdoor.

That is why the proof dispute works as a stress test. It is a low-stakes, highly visible case where a smart and motivated group of people can check the model’s homework against a precise ground truth. Most domains offer no such luxury. If we cannot get provenance right where the answer is a formal object that either holds or does not, we will not get it right anywhere the truth is fuzzier and the checkers are fewer.

There is a second-order cost too. Every unresolved dispute teaches researchers to stop sharing early work with these systems, which is the opposite of what a lab claiming to accelerate science wants. Trust is a supply chain. Break it in the visible cases and it thins out in the ones nobody is watching.

The incentives point away from attribution

None of this is an accident of engineering. The incentives are aimed the wrong way, and they are aimed there on purpose.

“Our model solved a hard open problem” is a recruiting tool, a fundraising line, and a benchmark headline all at once. Announcing first is worth real money. Slowing down to trace which human work fed the result is worth nothing to the announcement and might undercut it. A model that says “I produced this, and here are the three papers it leans on, one of them unpublished and shared with me during a private evaluation” makes a worse press release and a better scientific instrument. Companies are optimizing for the press release.

Watch what a lab measures and rewards, not what it writes in its safety blog. When the public claims are about raw capability, gold-medal performance, problems solved, and the provenance work is a footnote or missing, you can see which one has an owner with a budget and which one is a value statement. Attribution slows the demo down. Nobody gets promoted for adding friction to the demo.

What real provenance would require

This is fixable, but not with a statement of principles. It takes plumbing, and most of the pieces already exist in early form:

  • Training-data manifests. A versioned record of what corpora went into a model, so “was this paper in training” has an answer instead of a shrug. Data cards and datasheets for datasets are the first draft of this idea.
  • Retrieval with citations. When a model pulls from a source at answer time, the way retrieval-augmented systems do, surface that source in the output. This does nothing for the trained-in case, but it makes a large class of answers traceable by construction.
  • Output credentials. Content provenance standards like C2PA already sign images and video with their origin and edit history. The same signing can mark machine-generated text and name the model that wrote it.
  • Holdout and contamination audits. Keep test sets out of training and prove it, so “solved the benchmark” means solved rather than memorized.
  • Third-party access. Provenance a company grants itself and reports on itself is marketing. Independent researchers need enough access to test the claims, the way security researchers get to test a lock instead of trusting the brochure.

Formal proof assistants deserve a note here, because people reach for them as the fix. Lean, Coq, and Isabelle can machine-check that a proof is correct, step by step, with no trust in the author. That is real and valuable. But correctness is not origin. A verified proof tells you the steps hold. It says nothing about who first found them, which is the exact question in a theft dispute. Verification and attribution are different tools, and only one of them is being built at speed.

If it is your work

For a researcher who thinks a model reproduced their result, a few concrete moves matter more than the argument online.

Timestamp everything, publicly and early. arXiv, a DOI, a dated git history, a preprint server. The person who can show a dated artifact from before the model’s training cut-off holds the strongest hand, because provenance you keep beats provenance the lab failed to.

Assume anything you hand a system can come back out. If you share unpublished work in a company’s evaluation, a fine-tune, or even a long API session, treat it as potentially training data. Ideas you feed a model can reappear with no memory of you attached.

File specifics, not vibes. “This lemma, this construction, this ordering matches my Section 3” travels further than “they stole my proof.” You are supplying, by hand, the log the system never kept.

Ask for the manifest. Demand to know whether your paper was in the training set and whether any attribution method was run against the output. The answer, including an honest “we cannot tell,” is itself information about how the system was built and how seriously the builder takes the question.

The dispute will get argued as a story about one company and one result. It is really a question about whether the machines now producing knowledge can account for where that knowledge came from. Today they mostly cannot, and the people building them have little reason to change that until someone with authority makes provenance a requirement instead of a virtue.

Share

Keep Reading

Stay in the loop

New writing delivered when it's ready. No schedule, no spam.