Why ML Research Agents Don't Overfit: Winning Strategies Compress

Textbook machine learning warns that reusing a held-out set will burn you. Check your model against it, tweak, recheck, and repeat, and that set stops being unseen, so strong scores on it stop predicting real-world performance. Benchmark-driven research runs exactly this loop at community scale: everyone hill-climbs the same fixed benchmarks for years. By the theory, the leaderboards should be crowded with models that ace the benchmark and flop everywhere else. They aren’t. When researchers build fresh test sets for old, heavily reused benchmarks, the reported gains mostly hold up. Amazon researchers set out to explain why, using LLM-based research agents as a stand-in for the human community you can’t reset and rerun.

Their answer, laid out in the paper ‘What fits (into few tokens) doesn’t overfit,’ rests on a formal version of Occam’s razor. A hypothesis you can describe in very few bits, far fewer than it would take to memorize the training data, cannot have cheated: it lacks the room to record the answers, so good training performance means it captured real structure. Winning ML strategies turn out to be tiny in this sense. A project might generate thousands of benchmark scores, but the recipe that survives is a short list of familiar choices, such as an architecture family, an optimizer, a learning-rate schedule, and a couple of hyperparameters. Large language models sharpen the point because they act as powerful decompressors. Hand one a terse, expert-to-expert description and it fills in the standard tooling and defaults from its own world knowledge, none of which counts against the Occam bound since it never depended on the training data.

To test the idea, the team lets an explorer agent chase validation performance over hundreds of rounds, the same setup that ought to overfit, then squeezes the resulting strategy through a compression bottleneck to see how well it transfers. The framing matters beyond agents: it reframes a decade-old mystery about why benchmark chasing produced genuine progress, and it suggests that description length, not the length of the experimental transcript, governs whether iterative optimization generalizes.