Swapping Claude Opus for GPT-5.6 wasn't a drop-in — here's what broke

Ploy, which runs an AI agent that builds and edits real marketing websites, has made OpenAI’s newly released GPT-5.6 Sol the default model in every workspace, displacing Claude Opus 4.8. The company says Opus had held the top slot for four months of frontier-model testing before GPT-5.6 finally beat it on their own evals: pages finished in less than half the wall-clock time, cost 27% less, used roughly half the output tokens, and scored at or above the incumbent on completed work. Notably, GPT-5.6 writes far leaner code — one matched build produced a 2,508-character stylesheet versus Opus’s nearly 18,000-character equivalent.

The real lesson is that a ‘universal’ LLM SDK (Vercel’s AI SDK) hid how deeply the whole stack had specialized around one provider. Roughly a third of the initial cross-model failures came from the eval harness, not the model: tool-call budgets sized for Opus’s sequential style choked when GPT-5.6 fanned out parallel calls, and the executor didn’t support the batched file reads GPT-5.6 leans on. A silent default minScore of 1.0 was also failing hero pages that scored 0.98. Ploy’s blunt takeaway: triage the traces before you trust a challenger model’s pass rate, or you’re just grading it on how well it mimics the old one.

Two model-specific quirks did real damage. GPT-5.6 emits every optional tool parameter with invented-but-plausible values (offset: 0, a zeroed UUID), which the file-read tool treated as genuine — leaving 52–64% of reads silently returning empty while reporting success. Prompting and OpenAI’s strict mode didn’t help; the fix was a schema transform at the provider boundary that makes optional fields required-but-nullable, then strips the nulls before validation, cutting empty reads to zero and tool calls by ~30%. Prompt caching was the other trap: GPT-5.6 dropped partial-prefix caching entirely, so before Ploy rebuilt its cache strategy the model looked about 50% more expensive than Opus — a configuration artifact, not a pricing one. The piece is a useful reminder that migrating between ‘equivalent’ frontier models is an engineering project, not a config change.