Two Kinds of LLM Inference Optimization: Trading Off vs. Pushing the Frontier

Inference engineers borrow the economist’s “efficient frontier” to reason about serving tradeoffs — chiefly latency versus throughput (and therefore cost), but also quality versus efficiency and reasoning depth versus speed. The core argument is that optimization techniques fall into two camps: those that simply move a deployment to a different point along a fixed frontier by trading one factor for another, and those that push the whole frontier outward, creating extra efficiency you can then spend on lower latency, higher throughput, or both. Crucially, that frontier is jagged rather than smooth — small configuration changes can produce outsized effects, so the right settings for a given traffic pattern usually have to be found empirically through sweeps.

On the tradeoff side, batch size is the clearest lever: small batches give great per-user latency but poor cost-per-token, while larger batches invert that. Parallelism strategy matters too — Tensor Parallelism lowers latency (leaning on fast NVLink for its heavy all-to-all communication), Expert Parallelism can help both goals depending on its width, and Attention Data Parallelism replicates attention layers to raise throughput at the cost of per-request speed. Quantization is a hybrid case: dropping weight, activation, or KV-cache precision improves both latency and throughput, but it opens a separate quality-versus-efficiency frontier where microscaling formats like MXFP4 and NVFP4 can deliver large gains with little quality loss.

The frontier-pushing techniques are the ones that make headlines, and they compound — doubling performance in hardware and again in software yields a 4x gain. These include CUDA kernel and runtime optimizations that cut the resources needed per token, speculative decoding (EAGLE-3, DSpark, DFlash) that skips forward passes and shines on predictable workloads like code generation, and prefill/decode disaggregation, which splits the two inference phases onto dedicated workers so each can be tuned independently and their ratio matched to real traffic.