Databricks built its own coding-agent benchmark — and open models made the frontier

Databricks built an internal benchmark from its own recent pull requests to figure out which coding agents actually perform best on its multi-million-line, ten-plus-language codebase. Public benchmarks like SWE-Bench weren’t good enough: their tasks leak into training data and don’t reflect a stack spanning Scala, Go, Rust, TypeScript, Bazel, and Protobuf. By reconstructing well-specified tasks from human-written PRs that carried strong test suites, the team could measure both quality and real end-to-end cost with confidence they wouldn’t degrade their own developers’ workflows.

The results clustered models into three capability tiers and pushed back on some common assumptions. The Pareto frontier of quality-per-dollar now includes OpenAI, Anthropic, and open-source models — no single vendor covers it. Open-weight GLM 5.2 landed in the top tier, statistically tied with Opus 4.8 on quality ($1.28/task vs $1.94) and matching positive internal pilot feedback, making it a candidate daily driver. Crucially, per-token price is a bad proxy for task cost: Sonnet 5 is cheaper per token than Opus but cost more per task because it read more and burned ~1.9x the tokens to finish, while scoring lower.

The harness mattered as much as the model. Running the same model at the same reasoning effort through a lightweight harness (Pi) versus native tooling (Claude Code/Codex) swung cost per task by more than 2x with no quality difference, mostly because Pi fed roughly 3x less context per turn and kept a tighter working set. The practical takeaway for Databricks: stop defaulting every task to the most expensive model, route routine work (config flips, flag toggles) to cheaper Haiku- and GPT-5.4-Mini-class models, and invest in tooling (Omnigent) that makes swapping models and harnesses seamless.