Goodhart's Law at Scale: How LLMs Broke the Proxy Measures of Knowledge Work

Knowledge work has always been hard to evaluate directly, so reviewers fall back on proxy signals — clean prose, polished formatting, code that reads cleanly on a skim. Those proxies worked because producing surface polish used to require the same care that produced underlying quality. The correlation held well enough to keep incentives roughly aligned with real output.

LLMs sever that correlation. A model can generate a market analysis that mimics a top-tier consultancy deliverable, or thousands of lines of plausible-looking code, without any of the substantive reasoning the surface implies. Reviewers then use other LLMs to audit the output, surface issues get patched, and the ritual of work continues with none of the underlying rigor. The training objectives reinforce this — RLHF and next-token prediction optimize for output that looks like quality output, not output that is true or useful.

The result is a self-reinforcing loop: workers rationally optimize for the metric they’re judged on, that metric is now trivially gameable, and the volume of generated text outpaces anyone’s ability to read it carefully. The piece frames this as Goodhart’s law industrialized — billions spent building infrastructure that produces a convincing simulacrum of knowledge work rather than the work itself.