Claude optimizes what it measures

A profiler that reports a 2.3-second p95 latency is worth more to Claude than any instruction you can write about making a system faster. Optimization is downstream of measurement, without exception. Claude can cut a latency, reduce a token count, or lower a per-request cost only when the number is placed in front of it as data it can reason over. Strip the number out and you are asking a probabilistic system to hit a target it cannot see. It will still respond. It will restructure your code, swap a model, add a cache, and describe each change in confident language. None of it will be anchored to whether the system actually got faster, because nothing in the loop ever told it what faster meant.

The capability here is real, but it is narrow, and the narrowness is the whole point. Claude does not have a private view of your production metrics. It has whatever is in its context window. Give it a measurement and it can reason about the gap between current and target, propose a change, and check that change against the same measurement. The useful part is not the model’s intelligence in isolation. It is the closed loop: measure, change, re-measure, keep what moved the number, discard what did not. Claude is good at running that loop quickly and repeatedly. It is bad at running it blind.

So the practical claim is simple. If you want Claude to make a workflow faster, your first task is not a better prompt. It is instrumentation. Make the metric visible, define what faster means numerically, and give Claude a way to apply a change and read the result. A team that spends an afternoon wiring latency and cost into a form the model can consume will get more out of Claude than a team that spends a week refining the wording of a request. The measurement is the interface. Everything else is downstream of it.

Understanding why this holds means looking at how a model actually operates on a number. To Claude, a latency figure is not a fact about the world. It is a token in context or it does not exist. A trace sitting in your observability tool, a cost line in a billing dashboard, a duration logged to stdout - none of these are available to the model until they are captured and fed back in a form it can read. This is why measurement has to be built as an input, not treated as background truth. The workflow has to emit the metric, something has to capture it as structured data, and that data has to re-enter the model’s context before any optimization is possible. Skip any step and Claude is guessing.

The optimization itself is empirical, not analytical. Claude does not compute the optimal configuration of your pipeline the way you would solve an equation. It hypothesizes and tests, which is exactly how real performance work gets done. Nobody reasons their way to a 40 percent speedup from first principles. They measure a baseline, change one thing, measure again, and follow the numbers. Claude fits into that method rather than replacing it. The advantage is throughput on the loop: if the measure-change-verify cycle is wired properly, Claude can run many iterations that a human would run slowly, each one grounded in a fresh reading rather than a recollection of what should have worked.

That means the system around the model matters more than the model. Four components have to exist. A metric contract that states what you measure and in what units - p95 latency in milliseconds, cost in dollars per thousand requests, tokens per response. An instrumentation layer that produces those numbers reliably on every run. A feedback channel that puts the numbers back into Claude’s context after each change. And a verification step that re-runs the measurement so an improvement is confirmed, not asserted. The model stays probabilistic. The scaffolding around it - the measurement, the capture, the verification - has to be deterministic. That contrast is the design. You wrap deterministic control around a non-deterministic optimizer, and the measurement is the control surface.

The most common failure is asking Claude to make something faster without ever giving it a number. The output looks like work. It rewrites functions, introduces batching, suggests a smaller model, adds a retry policy. Every suggestion is plausible and some may even help, but none are verified, because there was no baseline to improve against and no reading to check the result. You end up with changes that feel like optimization and a system whose actual performance you never measured before or after. Plausible is not the same as faster. Without the number, you cannot tell the difference, and neither can the model.

The second failure is trusting the model’s own account of what it did. Claude will tell you a change reduced latency. It is describing what the change was designed to do, not what it observed happening, because in most setups it never observed anything. A model reporting success is not a measurement. Only re-running the metric is a measurement. The same trap appears when teams optimize a single dimension in isolation - cutting latency while cost silently triples, or shrinking cost while quality degrades below the point of usefulness. A real optimization loop carries the full set of metrics as constraints, so a win on one axis cannot quietly become a loss on another. One number is not enough. You need the number you are chasing and the numbers you refuse to sacrifice.

What has changed is that this loop is now something you can build and hand off, rather than run manually. Tool use lets Claude call a profiler, read a trace, and pull a cost figure directly. Structured outputs let it return proposed changes in a form a pipeline can apply without a human parsing prose. Evaluation harnesses let a change be scored against a fixed measurement automatically. Put those together and Claude moves from advisor to operator inside a measured loop - it can read the metric, propose the change, apply it, and re-read the metric, closing the cycle without a person in the middle of every step. That shift is the real development, and it comes with a condition that is easy to miss. None of it works unless you build the loop. The capability to optimize what it can measure is genuine. Whether Claude can measure anything at all in your system is a decision you make when you wire it up.

The loop that works has four moving parts, and none of them is the model. It starts with a metric contract written as a schema, not a sentence. A single object with named fields - p95_latency_ms, cost_per_1k_usd, tokens_per_response, quality_score - each tagged with a direction and a bound. The target metric is the one you are minimizing. The rest are guardrails with a floor or a ceiling: quality no lower than 0.80, cost no higher than the current bill. That schema is the contract in both directions. It is what your harness writes after every run and what Claude reads before every proposal. If the metric is not in the schema, it is not in the loop, and the model cannot reason about it.

The second part is an instrumentation layer that emits that object on every run, reliably, without a human formatting anything. Wrap the pipeline in a benchmark runner that executes a fixed set of representative requests - say 200 real queries pulled from production - records each duration, sums token usage into a cost figure, and scores every output against a held-out eval set. The fixed input set is not optional. Two configurations measured on different traffic are not comparable, and a latency drop that came from easier questions is not an optimization. Same requests, same eval set, every run. That is what makes the number mean something.

The last two parts close the cycle. Expose the runner to Claude as a tool and require proposals as structured output - a config diff, a parameter change, a model swap - not prose describing a change. The harness applies one proposal to an isolated copy, reruns the same 200 requests, and hands back a fresh metric object. Claude compares it to the baseline. The harness, not the model, decides what happens next: keep the change if the target metric moved and no guardrail was breached, revert it otherwise, and record the result either way. One change at a time, so attribution stays clean. The model proposes; deterministic code applies, measures, accepts, and reverts. Keep a human at the merge gate if the system is in production, but that human is approving verified results, not judging plausible descriptions.

A retrieval-augmented support assistant makes the shape concrete. Baseline, measured on 300 fixed questions: p95 latency 2.3 seconds, cost 4.10 dollars per thousand requests, answer quality 0.82 on the eval set. The target is latency. The guardrails are a quality floor of 0.80 and a cost ceiling at the current 4.10. Those three numbers go into the schema, the runner produces them on demand, and Claude gets read access to the runner and write access to the pipeline config. Nothing about the loop is clever. It is plumbing, and the plumbing is the point.

The loop runs one change at a time. First, Claude notices retrieval and metadata lookup run sequentially and proposes parallelizing them; the rerun gives p95 1.9 seconds, cost unchanged, quality 0.82 - kept. Next it cuts the reranker candidate pool from 50 to 20; the rerun gives 1.6 seconds, cost down to 3.70, quality 0.81, still above the floor - kept. Then it swaps the generation model for a smaller one; the rerun gives 1.2 seconds and 2.40 dollars, but quality drops to 0.76. That breaches the floor, so the harness reverts it automatically and logs the attempt. This is the single-axis trap caught in the act: on latency alone, that change looked like the biggest win of the run. Finally it trims the system prompt from 1,800 to 900 tokens by removing redundant instructions; the rerun gives 1.1 seconds, 2.10 dollars, quality back at 0.82 - kept.

End state: 2.3 seconds down to 1.1, 4.10 dollars down to 2.10, quality held at 0.82. Latency roughly halved and cost roughly halved, with no silent regression, because every accepted change was verified and the one bad change was caught by a number instead of by a reviewer’s intuition. The instructive part is the model swap. Claude proposed it in good faith and would have described it as a large speedup, and it was - the description was accurate about the latency and silent about the quality. Without the guardrail and the re-measure, it ships as a 48 percent latency cut and a quality regression nobody recorded. The intelligence that mattered in that moment was not the model’s. It was the harness refusing an unverified win.

Optimization is downstream of measurement, and the leverage lives in the instrumentation, not the model. A team’s real work here is building the metric contract, the fixed-input runner, the feedback channel, and the verification gate. Once those exist, the model is close to interchangeable - any capable one can run the measure-change-verify loop, because the loop, not the model, is doing the optimizing. That inverts where most teams put their effort. The afternoon spent wiring latency, cost, and quality into a schema the model can read returns more than the week spent rewording the request ever will.

Carry every metric that matters as a constraint, not just the one you are chasing. A loop that optimizes latency in isolation will trade away cost or quality without noticing, because a number you did not put in the schema is a number the system is free to sacrifice. The guardrails are what turn a fast pipeline into a fast pipeline you can still ship. And verify every change against a fresh reading - a model reporting success is describing intent, and only re-running the metric tells you what actually happened.

Claude can make faster only what you let it measure. That is the whole capability and the whole limit in one sentence. Give it a number, a way to change the system, and a way to read the number again, and it will grind the loop faster than a person can. Give it a request with no number behind it and it will hand you confident prose and an unchanged system. Whether the capability is real in your workflow was never a question about the model. It was decided the moment you either built the measurement or did not.

---

Contains a referral link.