Cognition's FrontierCode benchmark grades AI code on mergeability, not just correctness

Cognition has launched FrontierCode, a benchmark that evaluates whether AI-generated code would actually be merged by human maintainers rather than just whether it passes unit tests. Built with 20+ open-source maintainers from repos like Celery, Budibase, uppy, and Mattermost—each investing 40+ hours per task—the benchmark grades end-to-end quality including test design, scope discipline, style, and adherence to codebase conventions. Cognition claims an 81% lower false positive rate than SWE-Bench Pro, achieved through rubric grading, adversarial QC, and manual review of every task by a researcher.

Results suggest frontier models still struggle with production-grade code quality. On the hardest 50-task Diamond subset, Claude Opus 4.8 leads at just 13.4%, with GPT-5.5 at 6.3% and Gemini 3.1 Pro at 4.7%—though GPT-5.5 uses roughly 4x fewer tokens, giving it a better cost-intelligence ratio. The gap widens for open source: Kimi K2.6 tops out at 3.8% on Diamond. On the easier Main and Extended sets, Opus 4.8 reaches 34.3% and 51.8%.

The design choices reflect a critique of existing benchmarks: prompts are deliberately terse (a third the length of SWE-Bench Pro’s), tasks are drawn from multi-PR chains and freeform requests rather than scraped single PRs, and difficulty scales through rubric depth rather than patch size. The implicit argument is that functional correctness is now table stakes, and benchmark saturation has been masking how far models remain from the subjective judgment a tech lead applies during review.