Tuning Tokio: Yield for Latency, Batch for Throughput, Fear the Mutex

Writing async Rust that performs well on Tokio comes down to trade-offs rather than fixed rules, and the first discipline is skepticism. Long polls—stretches where a task runs without yielding—are everywhere in real applications and often harmless, so the author argues you should start from a user-facing metric you actually want to improve rather than hunting for red flags. Most performance problems turn out to live in application logic or the seams between distributed components, not in Tokio itself. When Tokio is implicated, the schedule-latency histogram (the gap between a task becoming runnable and actually being polled) is the most useful symptom to watch.

The core tension is fairness versus efficiency. To cut latency, yield more often: a server handling pipelined requests (Redis-style) can drain a buffered pipeline without ever returning to the runtime, starving other connections—inserting an explicit yield after each request can drop latency roughly tenfold. To raise throughput, do the opposite and batch, because every runtime interaction has a cost. tokio::fs is a repeat offender since, without io_uring, each filesystem call hits the shared blocking pool; grouping blocking work into larger segments, or using a dedicated OS thread, amortizes that overhead. The same logic applies to task spawning—putting a 10-microsecond unit of work on its own task usually costs more than it saves.

Two structural warnings round it out. Some runtime resources are global bottlenecks: the blocking pool and the global task queue require shared coordination, and spawn_blocking can become visible in flamegraphs at high rates (the author saw degradation around 50,000 blocking tasks per second on a 32-core host). Mutexes are the sharpest footgun—one contended lock, such as a metrics registry held during a flush, can stall every worker and defeat work-stealing entirely. The guidance is to keep critical sections tiny, avoid RwLocks (whose atomics contend even on reads), and never hold a lock across I/O or an await point.