Claude optimizes what it measures

This August, Anthropic ran a two-week sprint that made claude.ai and its desktop app about three times faster, and merged more than three thousand changes along the way without a single rollback or customer-facing incident. The work ran in a Slack channel with multiple engineers and an internal Claude model, roughly comparable to Opus 5.5, working in every thread: Claude found bottlenecks, built benchmarks, shipped the improvements, watched the deploys, and read the field data afterward, while the engineers set goals, made tradeoffs, and approved every change. At the 75th percentile, time to a typeable page on a fresh load of claude.ai fell from 3.1 seconds to 0.55, starting a Claude Code session from 0.8 to 0.3, and loading a Cowork cloud session from 2.6 to 0.73.

The team started by asking Claude to analyze usage data through a Datadog MCP server. It picked the four journeys that cover 95% of activity (launching the app, starting a conversation, loading an old one, and sending a message), which came to thirteen distinct measurements across web and desktop. They added instrumentation until each measurement was comparable: start at a user interaction, stop when the result rendered, separate client work from server work. Claude estimated each planned project’s impact in milliseconds, the team summed those into sprint targets, and twelve of the thirteen targets were hit by day three.

Measuring is now the first step, not the zeroth

The useful observation from the sprint is about sequencing. Measurement used to be the thing you did before the work: add a metric, wait for data, then start to understand the problem. Here it became the first move of the optimization itself. As soon as Claude had a number to beat, it would start driving the number down, so the most valuable thing the humans could do was find more things to count.

That turned into a specific pattern. Every new benchmark had two jobs. First, it was a metric Claude could move in the lab, without waiting for a deploy and a field read. Second, it became a guardrail in CI whose number could only ratchet down. On two hot paths (the routine that assembles a conversation’s message tree, and a scanner for status lines in Claude Code output) Claude profiled with Valgrind, found that a quarter of the first path’s instructions were megamorphic dictionary lookups resolving the same message ID three times, and cut instruction counts by 48% and 31%. The team checked in two ratchets: any PR that raises those counts fails CI, and a daily job lowers each ceiling whenever the count drops. The win is locked in and can only improve.

Give that loop more counters and it finds more work. A React hook census turned up 6,900 hooks and 900 store subscriptions re-rendering on every keystroke in the composer. A style-recalc count found a single :root:has() selector adding 24 milliseconds to every DOM change. Tracing code paths after first paint surfaced a leftover location.reload() causing half a million hidden reloads a day that no load metric could see. A CPU sweep found that highlighting a finished code block could freeze the page for about a second: any non-Latin-1 character in the reply, an em dash or a curly quote, pushed V8 to store the whole string as UTF-16 and put every syntax-highlighting regex on its slower two-byte path. The fix was twenty lines that copy each block into a one-byte string first.

The hard part is picking metrics that mean something

An optimizer will climb whatever you point it at, including a proxy that doesn’t track the thing you actually care about. The team treated this as the main risk. Wall-clock time is what users feel, but it is noisy and milliseconds are too flaky for a CI gate. Instruction counts are deterministic and make a clean gate, but only matter if they move wall-clock. So before trusting a lab metric, they made Claude prove it: “we’ll unship the benches for any candidates that cannot prove that.” On the two instruction-count paths, the 48% and 31% cuts came with wall-clock drops of 78% and 44%, which earned the ratchets. Benchmarks that were flaky, or that didn’t correlate with user latency, were thrown out rather than left for Claude to climb the wrong hill.

Bespoke metrics did the same job for problems the standard ones missed. Sidebar rows popping in after load felt janky, but Cumulative Layout Shift scored each shift at about 0.008, well under the 0.1 “good” threshold. Reading the underlying Layout Instability API directly, Claude built a telemetry event that tagged each layout shift with a named region and phase, and an integration test that held the sidebar’s data until after first paint and failed on any shift in any named region. It went red 20 of 20 on main and green 20 of 20 on the fix. In the field, that event showed 31% of web loads moving something after the page was usable, and Claude worked through the causes one by one.

The guardrails came before the agent

Because nearly everything touched was a hot path, the safety mechanisms were in place before the pace picked up. Every PR went through automated review plus at least one human approval. Unit tests landed before the optimizations they protected. Anything user-visible shipped behind a short-lived feature flag. On the busiest days more than two hundred changes landed, and about a third of PRs added telemetry or guardrails of their own.

Three things are worth copying if you run an agent this way. Flag hygiene is one: the sprint introduced nearly two hundred flags, classified each as a kill switch or a ramp, and cleaned up more than half before the end. An agent that ships fast accrues flag debt fast, so cleanup got its own coordinating thread. Incremental rollout is another: employees first, then 1%, then everyone. That caught a layout shift none of their metrics saw, where Chrome’s speculative prerender painted claude.ai at a managed browser’s slightly shorter new-tab height and resized a tenth of a second later. And the humans kept three jobs the model would not do well on its own: pushing it past the first target it hit, ruling on user-perceptible tradeoffs with before-and-after recordings, and keeping each of the 150-plus concurrent threads narrow enough to stay useful. One 900-line PR was closed with a single line: 2 milliseconds per send was not worth the build-plugin complexity.

Performance wins decay in a fast-moving codebase, which is why the ratchets matter more than the individual fixes. The static composer shows users an HTML copy of the page immediately and lets React paint on top of it, so a one-pixel drift breaks the effect; it is kept honest by a jsdom test that fails if the static markup and the real component diverge, an alignment suite that checks fourteen viewport sizes to within a pixel, and a keystroke test that fails on any lost or reordered key through the handoff. The 95th percentile, long conversations, and other journeys still have room, but the gates keep the two-week gains from eroding on their own.

---

Contains a referral link.