Same task, different bets

Across roughly 17,000 agent runs, Claude, Codex, and Cursor did not converge on the same tools. Handed near-identical coding tasks, they assembled measurably different sequences of actions to reach an answer. Claude tended to read and search the codebase before touching a single line. Codex moved to edits faster, with fewer exploratory calls in front of them. Cursor leaned hardest on editor-resident context and made narrower, more localized changes. Same tasks. Different strategies. That gap is the finding, and it is not a leaderboard.

The practical takeaway is blunt: tool choice is a proxy for how a model decides to work, and how a model decides to work is what determines whether it survives inside a pipeline. A benchmark score tells you whether the model can produce a correct patch in isolation. The distribution of tool calls tells you what it will actually do when you point it at a real repository with tests, build steps, and side effects. Those are different questions, and most teams only ask the first one. When Claude spends its first several actions on read and grep, it is buying grounding before it commits. When Codex skips ahead to an edit, it is spending fewer tokens and less wall-clock time, and accepting more risk that the edit is aimed at the wrong place. Neither is wrong. They are different bets, and the bet is visible in the tool trace long before it shows up in the output.

Read the runs and a shape appears. Exploration-heavy agents front-load reads, searches, and directory listings, then edit late and often verify by running something. Action-heavy agents compress that phase, edit early, and lean on the model’s internal picture of the code rather than re-checking the ground truth. Editor-integrated agents inherit context from the open workspace and therefore search less because the relevant files are already in view. If you are wiring one of these into an automation pipeline, the tool distribution is the specification you are actually buying. It predicts token cost, latency, blast radius, and how much cleanup a human has to do afterward far better than any single-number score does.

Underneath the surface, every one of these agents is running a policy over an action space. The action space is small and boring: read a file, search for a string, list a directory, edit, run a command, run the tests, report done. A run is just a sequence of choices over that set until the agent decides it has finished. Measure 17,000 of those sequences and you are no longer looking at anecdotes. You are looking at the policy each system has learned or been prompted into: which action it reaches for first, how long it explores before it acts, when it decides to verify, and when it decides to stop. The distribution is the behavior. The individual run is noise.

The reason the policies differ is that each system is optimizing against different constraints, whether or not anyone stated them out loud. Reading and searching cost tokens and latency but reduce the chance of editing the wrong thing or hallucinating an API that does not exist in this codebase. Editing early saves those tokens and gets to an answer faster, at the cost of grounding. Running tests after an edit is a verification step that catches errors the model cannot see from the source alone, but it adds a slow, sometimes flaky dependency to every run. Claude’s heavier exploration is a correctness-first prior. Codex’s earlier edits are a throughput-first prior. Cursor’s reliance on open-editor context is a prior that says the human has already narrowed the problem, so the agent should act inside that frame rather than re-derive it. These are the classic LLM engineering trade-offs - grounding versus speed, verification versus cost, exploration versus exploitation - expressed not in a design doc but in which tool the model calls next.

This is why aggregate tool counts are more honest than they look. A model that averages a dozen reads before its first edit is telling you it does not trust its own memory of the repository, which is usually the right instinct in an unfamiliar or large codebase. A model that edits after two reads is telling you it will move fast and occasionally aim at the wrong file. A model that rarely runs the tests is telling you the verification burden lands on you, downstream, in your CI or in a human review. None of that requires you to read the model’s weights or its system prompt. It falls straight out of counting what the agent did across thousands of runs. Once you see tool choice as a revealed policy rather than a cosmetic detail, the differences between these three stop being trivia and start being architecture decisions you are inheriting.

The first mistake is treating these agents as interchangeable because they can all solve the same demo. On a clean, well-scoped task, all three will often land a correct change, and that similarity hides everything that matters. The differences show up under load: a large repo where blind edits miss the right file, a task with subtle cross-file dependencies where skipping exploration produces a plausible but wrong patch, a run where the tests would have caught the error if the agent had bothered to run them. If you pick an agent from a two-minute screen recording, you are selecting on the easy case and ignoring the tail, and the tail is where automation pipelines actually break.

The second mistake is reading the tool counts as a virtue score in either direction. More tool calls are not smarter, and fewer are not more efficient. Exploration that does not change the final edit is wasted latency and wasted tokens. Speed that skips the read the task actually needed is just a faster way to be wrong. The number itself means nothing without the outcome attached to it. The question is never whether an agent explores a lot or a little. It is whether its policy matches the shape of your work: how large your codebase is, how coupled your changes tend to be, how expensive a wrong edit is to catch and revert, and how much verification you can afford to run on every single execution. An exploration-heavy policy is a bargain on a sprawling legacy repo and pure overhead on a tight, familiar module.

The third and most expensive mistake is choosing an agent from someone else’s benchmark instead of measuring the tool distribution in your own environment. Your repository, your test suite, your constraints, and your definition of an acceptable change are what determine which policy pays off. The 17,000 runs are useful because they show that the differences are real and stable, not because they hand you a winner to copy. The move is to run the same set of representative tasks through each agent in your own stack, capture the full tool traces, and look at where the calls cluster, where the edits land, and whether the agent verified its own work before declaring victory. That trace is the actual product you are integrating. Judge it as a system, not as a personality, and you will make a decision you can defend when it hits production instead of one you have to walk back the first time it edits the wrong file.

The workflow that survives contact with production starts by treating the tool trace as a logged artifact, not a debugging afterthought. Capture every tool call each agent makes with a run ID, a task ID, the arguments, and the result, and you can compute the revealed policy in numbers you can actually threshold. Five metrics carry most of the weight: reads and searches before the first edit, distinct files touched per run, lines changed per edit, whether a test or build command ran after the last edit and before the agent declared done, and total tokens and wall-clock per run. Run that instrumentation over a fixed set of representative tasks from your own repo and you stop guessing about behavior. You have the distribution, and the distribution is the thing you are integrating.

The place these pipelines break is rarely the model being incapable. It is the gap between the policy the agent runs by default and the guarantees your pipeline assumed but never enforced. An action-heavy agent that verifies in 40 percent of runs will happily report success on a patch that fails the suite, because nothing in its loop required it to run the suite. That is not a hallucination problem you fix with a better prompt. It is a missing control, and prompts do not reliably install controls. The same failure hides on the exploration-heavy side: an agent that reads twelve files before every edit is buying grounding you may not need on a tight module, and you pay for it in latency and tokens on every single run whether or not it changed the outcome. Both failures are structural, and both are invisible until you count.

So the move is to pull the parts of the policy you cannot leave to chance out of the model’s discretion and into a thin harness that wraps every agent behind the same interface. You run the tests, not the agent, and the exit code gates the done signal. You cap files touched per run and reject anything that exceeds the blast radius the task should have. You inject the relevant file contents into context before the first edit for the fast agent that will not fetch them itself. Then you match the guardrail to the measured weakness rather than applying one generic wrapper to all three. An exploration-heavy policy mostly needs a token and step budget so its reads do not run away. An action-heavy policy needs supplied grounding and a forced verification step. Editor-context agents need a check that the failing file is actually the right frame before they commit inside it. Deterministic control around a probabilistic policy is the whole job, and the distribution tells you exactly which controls each agent is missing.

Consider a platform team wiring an agent into a CI job that auto-fixes failing lint and type errors and small bug tickets across a 400,000-line monorepo. They pilot all three on 200 representative tickets and capture the full traces instead of eyeballing a few outputs. The numbers separate cleanly. The exploration-heavy agent averages around eleven reads and searches before its first edit, touches under two files, and runs the suite before reporting done in roughly 82 percent of runs. It costs more tokens and more wall-clock, but the patch that reaches human review is usually aimed at the right place and already passes local tests. The action-heavy agent reads about three times before editing, moves faster, and verifies in closer to 40 percent of runs, at roughly half the token cost and latency. On single-file tickets it is a bargain. On cross-file tickets it lands the edit in the wrong module more often, and those wrong edits slip through precisely on the runs where it skipped verification. The editor-context agent edits narrowly and searches least, assuming the file where the error surfaced is the file to fix. It is excellent on single-file lint and type errors and weak whenever the root cause lives somewhere other than where the symptom appeared.

The useful outcome is not crowning a winner. It is routing by ticket class based on what the traces showed. Single-file lint and type errors go to the fast, narrow policy with a mandatory test gate bolted on by the harness, since that is the one control it reliably lacks. Cross-file bug tickets go to the exploration-heavy policy, where the extra reads pay for themselves in avoided wrong-file patches. Every route gets the harness-enforced test run regardless of whether the agent would have run it on its own, because verification is now the pipeline’s responsibility and not the model’s mood. The measured result is stated in operational terms the team can defend: fewer wrong-file patches reaching review, a predictable cost envelope per ticket class, and reviewers spending their attention on the genuinely ambiguous fraction of tickets instead of catching blind edits the harness should have blocked. None of that came from a benchmark score. It came from counting actions and building around the counts.

Tool choice is a revealed policy, and the policy is the product you are actually integrating. A leaderboard tells you an agent can produce a correct patch in isolation. The tool distribution tells you what it will do to your repository when no one is watching, which is the only question a pipeline cares about. Measure it in your own stack, on your own tasks, and read the clusters honestly. More calls are not smarter and fewer are not leaner. The number means nothing without the outcome attached to it.

The harness is where the differences you measured become guarantees you can rely on. Once verification, blast radius, and grounding are enforced outside the model, the agent’s default policy stops being a risk you absorb and becomes an input you shape. You can run a fast, cheap policy safely because the pipeline supplies the caution it skips, or run a cautious one economically because the pipeline caps the exploration it overspends on. Either way the control lives in code you own, not in a probabilistic loop you hope behaves.

The hard part is accepting what you are choosing. You are not picking a model. You are inheriting a work strategy, a set of bets about grounding versus speed and verification versus cost that the vendor made and encoded into how the agent reaches for its next tool. The only way to know which strategy you bought is to count what the agent did across thousands of runs and to wrap the result in controls that cover its weaknesses. Do that and the decision holds up in production. Skip it and you will find out which bets you inherited the first time the agent edits the wrong file and reports that it is done.

---

Contains a referral link.