Real-SWE benchmark: AI coding agents flunk real enterprise codebases, failing ~72% of tasks

A team at Specific has released Real-SWE, a benchmark that drops frontier coding agents into private production codebases licensed from actual companies rather than the synthetic or expert-authored tasks most evaluations rely on. The tasks are drawn verbatim from work real engineers were paid to do — fixing invoice tax logic, migrating customers, wiring up external billing services — and they carry genuine business consequences. Because the code and its solutions never appear on the public internet, the benchmark is natively out-of-distribution; the authors argue that roughly 99% of enterprise code tokens are invisible to the models being tested. Evaluations run model-and-harness combinations together in isolated sandboxes, with verifiers borrowed or adapted from each codebase’s own test suite.

The results are unflattering. Agents failed the large majority of tasks regardless of how long they ran — about 71% of rollouts under ten minutes and 73% of longer ones — showing that more thinking time didn’t rescue them. The single most common failure mode was missing requirements outright: models struggle to absorb existing business logic and company-specific coding conventions, and they frequently skip stated needs or fail to verify their own assumptions. Instructions are deliberately terse, leaving agents to discover implementation details across code, infrastructure, and business tools like Postgres, Kubernetes, GitHub, Linear, and Slack, and a single change often spans multiple services.

The takeaway is that current models remain far from the bar real engineering organizations hold, even though the tested tasks are cheap to attempt — estimated rollout costs ran from $2.50 to $6.96. Beyond raw pass rates, Real-SWE is a pointed critique of how AI coding progress gets measured: benchmarks built to showcase interesting capabilities can overstate readiness for the economically meaningful, convention-bound, cross-functional work that actually fills an engineer’s day.