Your Local LLM Isn't Dumb — Your Inference Stack Is Diverging From the Lab's

The gap between a model’s advertised benchmarks and how it feels on your own hardware isn’t imagination — it’s the inevitable result of running the same weights through a different stack. The lab that publishes a model hosts it on its own hardware and software, and every home setup diverges from that reference: mixed GPU generations with different instruction sets compute the math behind each next token slightly differently, and a nightly vLLM image can pull in hundreds of separate packages, each with its own quirks. Two systems running identical weights will still take different paths through that mountain of code, and those small numerical differences compound into different token choices.

To quantify the drift, the author leans on both practical and mathematical measures. On the practical side: run a variety of real benchmarks that match your actual workload rather than cranking temperature to zero and pasting in three prompts, since zero-shot tests poorly predict long-context, tool-calling agentic work. On the math side, KL divergence measures how far a model’s output probability distribution has moved from a chosen baseline — but it’s directional and only meaningful if the author discloses the reference checkpoint, runtime, evaluation data, calibration set, context lengths, and aggregation method. The piece warns against trusting suspiciously low KLD figures on quant model cards, and notes practical footguns like setting temperature too low, which can trap reasoning models in endless think-loops.

The first experiment isolates a single stage: the attention backend used during prefill. Running an official BF16 Qwen checkpoint on a Blackwell GPU with quantization and other optimizations disabled, the author replayed a real ~100k-token agentic workstream — deliberately chosen because it appears in no benchmark or training set, so no one could have tuned for it — while swapping only among FlashAttention 2, Flash Inference, and Triton. For the first several thousand tokens all three backends agreed on the greedy next token, but deeper into the long context they began flipping choices. The takeaway: the backend selection alone, before quantization even enters the picture, is enough to steer generation apart — and a divergence that starts as one different token is exactly where an eventual tool call could go wrong.