Why Your p50 Lies: Customers Live in the Tail of Your Latency

AWS engineer Marc Brooker explains a measurement gap that makes service operators and their users both correct when they disagree about speed. An operator measures a 100ms mean request time; the customer swears the typical wait is a full second. The cause is the inspection paradox: people don’t experience your latency distribution, they experience a time-weighted version of it, because the moments they spend waiting are disproportionately the slow ones. Formally, the latency a user feels equals the mean plus the variance divided by the mean — so the heavier your right tail, the further customer-perceived latency drifts above your reported average.

The same logic applies to reliability. An operator can boast a sub-minute MTTR while customers insist outages drag on for an hour, and both numbers are real. Brooker walks through an example where a 30-minute median time-to-recovery paired with a rare 10-hour worst case yields an MTTR just over an hour, yet customers experience a mean recovery closer to six hours. Retries and timeouts can sometimes mask slow requests (as long as no locks or exclusive resources are held), but nothing can hide a long recovery — outage tails land directly on users.

The practical takeaway is that tail behavior, not the average, dominates how humans perceive a service, which is why Brooker dislikes trimmed means and other summaries that discard the shape of the right tail. He uses a log-normal distribution in his interactive simulator purely for mathematical convenience, cautioning that it isn’t a good model for real latency or recovery data — he’d reach for a non-parametric approach in practice.