Why bigger server fleets get lower latency at the same utilization

Marc Brooker poses a deceptively simple queuing puzzle: take c single-threaded servers behind a load balancer, scale the offered traffic linearly so each server stays at 80% utilization, and give every request a one-second average service time. As c grows, does client-observed latency get better, worse, or stay flat? Intuition splits, but the answer is that mean latency falls and asymptotically approaches the bare one-second service time — the time requests spend waiting in queue trends toward zero.

The mechanism is classic M/M/c queuing theory, what teletraffic engineers call Erlang’s delay system. Erlang’s C formula shows that at a fixed utilization, adding servers sharply cuts the probability that an arriving request has to wait at all: a 5-server system at half load queues about 13% of requests, while a 10-server system at the same relative load queues only 3.6%. A Monte Carlo simulation confirms the win isn’t hiding in the tail — the median, p99, and p99.9 all improve at the same rate, so there’s no nasty surprise lurking in the percentiles.

The practical payoff is in cloud economics: a larger fleet buys you either better latency at the same utilization or higher utilization at the same latency, with no change in per-server throughput. Crucially, most of the benefit shows up at modest server counts, so this isn’t a perk reserved for hyperscalers. It’s one of the rare cases where scale makes a distributed system easier rather than harder. Brooker flags the usual caveats — the model assumes Poisson arrivals and exponential service times, and real-world service times look more log-normal — but the qualitative result holds, and the system stays stable as long as arrival rate stays below total service capacity.