Study Breaks Down Coding-Agent Harnesses: What Actually Moves the Needle

Autonomous coding agents live or die by their harness — the scaffolding that turns a model’s raw ability into sustained software-engineering work — yet harnesses are usually benchmarked as black boxes, obscuring which parts matter. Researchers isolated the question by holding the execution loop constant and varying three components independently: planning, action space, and context management. Running four models against SWE-Bench Verified and Terminal-Bench 2.1, they swept 176 matched configurations covering five context-management strategies, four context-window budgets, and focused ablations.

The results argue for tuning the harness to the model and the budget rather than bolting on machinery for its own sake. Context management pays off mainly as the context window shrinks, and almost all of its value comes from averting outright context-overflow failures rather than making the agent smarter. The most efficient recipe applies rule-based elision first, then LLM summarization; building in the ability to recover elided content adds complexity that models seldom exercise and buys no accuracy. Planning behaves differently across capability tiers — it props up accuracy for weaker models but mostly just cuts cost for stronger ones. Similarly, predefined tools help models with shaky bash skills, while bash-fluent models run fine on a bash-only interface at markedly lower cost, especially on command-line-heavy tasks.

Trajectory analysis ties the findings together: context management lets agents run longer without changing how they behave, planning changes where they decide to stop, and the action space dictates how finely code gets written. The upshot is a modular framework for evaluating harness components in isolation and a case for model- and budget-aware design over one-size-fits-all systems — useful guidance as coding agents move deeper into real engineering workflows.