The 150x model tax

Inference engineers work with two kinds of lever. One kind moves a deployment along a fixed tradeoff curve, trading faster per-user responses for lower throughput or the reverse. The other kind moves the curve itself, creating efficiency you can then spend on lower latency, higher throughput, or some mix of the two. Baseten borrows the term “efficient frontier” from economics to describe that curve. For models, a frontier model is the one offering the most intelligence at a given cost or size. In serving, the frontier is usually the tradeoff between latency and throughput, and throughput is what determines cost.

Both kinds of lever matter. Giving up per-user speed lets you build high-throughput, low-cost pipelines for batch work. Sacrificing throughput for speed makes sense when latency-sensitive users are willing to pay for it. The techniques below assume something like GLM-5.3 or Kimi K3 running agentic coding, with KV cache reuse and KV-aware routing already in place.

Levers that move you along the curve

The frontier is jagged, not a smooth line between outcomes. Small configuration changes can produce large shifts, and the cutoff points are unintuitive enough that you find them by running sweeps rather than reasoning them out.

Batch sizing is the most direct tradeoff. A batch is the number of requests processed concurrently. Token-level continuous batching removes the latency of waiting for a batch to fill, but the configured batch size still sets both per-user latency and total throughput. Small batches give excellent per-user latency and generate few tokens per GPU, so cost per token is high. Larger batches invert that result: worse per-user latency, better throughput, lower cost.

Parallelism is the next lever. Models in the hundreds of billions or trillions of parameters have to be spread across GPUs, and how you spread them favours either latency or throughput. Tensor Parallelism carries expensive all-to-all communication but lowers latency, because those operations run fast over high-bandwidth NVLink interconnects. Expert Parallelism can help both: a lower degree tends toward better latency, while wide EP across a full rack supports higher throughput. Attention Data Parallelism replicates attention layers for parallel computation, buying throughput at the cost of per-request speed.

Quantization sits in both camps. Running weights, activations, or KV cache values at lower precision improves both latency and throughput, so it pushes the serving frontier outward. It also opens a second frontier between quality and efficiency, and that one is especially jagged: large efficiency gains are available with little or no quality loss, particularly with microscaling float formats like MXFP4 and NVFP4.

Levers that move the curve

These compound, which is why they get the attention. Double performance from better hardware, double it again from better software, and you have a four-times improvement to allocate however you want.

Kernel and runtime optimization is the base layer. A CUDA kernel is a low-level function running one piece of inference, such as a matrix multiplication. Making individual kernels and the full forward pass faster means fewer resources per token, and those savings compound up the stack.

Speculative decoding guesses the tokens a model will generate, then validates the guesses. Early on this was itself a tradeoff: speculation was expensive, sequences were short, acceptance rates were low, so it only worked at small batch sizes. Techniques like EAGLE-3, DSpark, and DFlash still compete with the main model loop for resources, which caps maximum batch size, but they perform well enough, especially on code generation where output sequences are relatively predictable, that they deliver efficiency from skipped forward passes on top of the raw per-user speedup.

Disaggregation separates prefill and decode onto dedicated workers. For high-volume deployments that lets you tune each phase independently and adjust the ratio of prefill to decode workers to match the input and output sequence lengths and cache hit rates in your actual traffic.

Which lever you are pulling matters in practice. Batch size, parallelism degree, and reasoning level move you along the curve, so a gain in one dimension costs you another, and you only land on the right point by sweeping the jagged regions against your own traffic. Kernel work, speculative decoding, and disaggregation move the curve itself and compound with each other, so they hand you efficiency to allocate before any tradeoff is forced.