Cloudflare reclaims 100TB of RAM by rethinking consistent-hashing math

Cloudflare’s Performance team traced excessive memory use in Pingora Backend Router (PBR), the company’s internal load balancer, to structures inside pingora-ketama — its open-source consistent-hashing library. Consistent hashing maps both servers and cache keys onto a fixed integer output space (a conceptual ring), assigning each key to the nearest server. Because hash outputs are effectively random, a single hash per server produces wildly uneven load: with 100 servers the variation in each server’s share approaches 99%. The standard fix is to give every server many virtual points on the ring — NGINX and Pingora default to 160 per server — which averages out the imbalance and drops the coefficient of variation to roughly 8%.

That many-points-per-server approach is exactly what inflates memory. Multiplying 160 hash entries by large fleets of servers across every node builds up large in-memory tables, and at Cloudflare’s scale the overhead compounds into hundreds of terabytes. The ketama variant adds another wrinkle: Cloudflare weights request distribution by each server’s disk capacity rather than distributing evenly, so more storage means more allocated points.

By reworking how these hash points are represented in Rust, the team cut PBR’s footprint enough to free more than 100TB of RAM globally — on top of a separate 100TB the DNS team recovered the prior month. The takeaway is a familiar one for hyperscale operations: at millions of cores and petabytes of RAM, a targeted change to a single hot algorithm delivers savings that dwarf what any per-service tuning could achieve.