OpenAI says SWE-bench Verified is too broken to trust, pushes SWE-Bench Pro

OpenAI examined SWE-bench Verified—one of the most widely cited benchmarks for AI coding ability—and concluded it no longer produces a meaningful signal about real software-engineering capability. The problems are structural: many tasks ship with underspecified or ambiguous prompts, and their hidden test suites check for one specific implementation rather than functional correctness, so a valid solution can fail while the description and the grader disagree. Add data contamination and a small task pool (fewer than 800 items), and scores end up reflecting how well a model fits the benchmark’s quirks rather than whether it can actually do the work.

The deeper concern is downstream. Capability numbers from evals like this feed deployment and safety decisions, including those made under OpenAI’s Preparedness Framework, so a flawed benchmark can quietly distort safety cases and skew research priorities. It’s a textbook Goodhart’s Law trap: once a benchmark becomes the target, labs optimize for the metric instead of the underlying skill, and the metric stops measuring anything useful.

OpenAI’s recommended fix is to move the community toward SWE-Bench Pro, built to test longer-horizon, more realistic agentic coding tasks that are harder to game and better aligned with how engineers actually work. The broader takeaway for anyone tracking model progress: treat headline coding scores skeptically, and weight evaluations that reward genuine end-to-end problem solving over narrow test-case matching.