Sutton: Generative AI Can Be Novel or Good — Never Both at Once

Reinforcement learning pioneer Rich Sutton argues that generative AI trained purely by supervised learning is structurally incapable of genuine discovery. Borrowing the old reviewer’s joke — ‘the parts that are good are not novel, and the parts that are novel are not good’ — he contends that LLMs and image/video models produce quality only by mimicking their training data, and novelty only through randomness. The two never coincide in a single output. Useful novelty that departs from source material is, in his framing, what we already call hallucination.

Sutton’s diagnosis is that mimicry-based systems lack the three-step loop he calls Discovery: variation, evaluation, and selective retention — the same mechanism underlying natural selection, the scientific method, and reinforcement learning. Generative models have variation via stochastic sampling, but no runtime evaluation of what they generate, so nothing valuable can be selectively retained. Echoing earlier arguments by Donald Campbell and Daniel Dennett, he extends the point to modern ML: backpropagation and gradient descent alone contain no Discovery.

The implication matters most for using AI in science and mathematics, where creativity is the entire point. Sutton points to systems that do close the loop — AlphaGo’s move 37, AlphaZero, AlphaFold, AlphaProof, GT-Sophy — as proof that AI augmented with evaluation and selection can find things both novel and good. Today, that evaluation step is often supplied by humans picking the best of many generated outputs; for autonomous discovery, it must be built into the system itself.