The Hidden Chaos of OpenRouter: Same Model, Wildly Different Providers

The operator of Olly, an iMessage AI assistant that has handled over 18 million messages (a third on open models via OpenRouter), lays out the operational traps of routing to third-party inference providers. The core problem is that OpenRouter’s model IDs are an abstraction over roughly 20 different companies that host the same weights at their own precision, with their own optimizations and their own parsing bugs. On paper they serve identical models; in practice they behave like different models entirely. Per-provider benchmarks make this concrete — on DeepSeek V4 Flash, first-party scored 90% GPQA and 81% on the TAU-Bench tool-calling task, while another host of the same weights managed just 75% and 58%. For agentic workloads, that 20-point tool-calling swing is the difference between working and broken.

The failure modes go well beyond scores. Some providers of a supposedly vision-capable model can’t actually see images yet return 200 OK anyway; the reasoning-effort parameter is silently ignored by some hosts; and declared quantization (fp8 vs fp4) turns out to be a poor proxy for quality, so filtering on it just shrinks the fallback pool. Other quirks include tool calls leaking through as raw markup when a provider’s parser misses, ‘hollow’ 200 responses with null content and no usage object (StreamLake and later Together were repeat offenders), and per-provider disagreements on conversation-history contracts — SiliconFlow 400s if empty reasoning blocks aren’t passed back, while Baidu, Alibaba, and Cloudflare accept the same history fine.

The practical takeaways: benchmark from your production infrastructure rather than your laptop, since some hosts rate-limit by IP and will 429 prod while working locally. Treat HTTP 200 as ‘served,’ not ‘answered,’ and build client-side parsing and retry logic. And resist pinning a single provider for reliability — the author pinned three of the most dependable hosts for the top model, only to watch all three degrade into rate-limiting or drop the model entirely within two weeks, taking his product down with them.