Open-source Qwen3-TTS serving hits sub-50 ms latency at 1/25th the cost

Nari Labs rebuilt the serving stack around the open-weight Qwen3-TTS 1.7B CustomVoice model and pushed p95 time-to-first-audio below 50 ms while sustaining 10 requests per second on a single H100. In a five-minute Poisson open-loop benchmark against four other engines (vLLM-Omni, SGLang-Omni, VoxServe, and M*), theirs was the only implementation to stay under 50 ms at that load, holding below 100 ms even at 20 RPS. The economics are the headline: roughly $2 per million characters at full utilization, versus $100 for ElevenLabs V3 and $49 for Cartesia Sonic 3.5.

The gains come from treating latency as a scheduling problem rather than tuning modules in isolation. Qwen3-TTS runs as three stages—a Talker, a Code Predictor, and a causal Codec—and most implementations fuse the first two while decoding separately. Nari instead exposes all three as independently schedulable tasks under one scheduler, which can reorder work by urgency: prioritize requests that haven’t yet produced audio, and treat active streams as urgent only as they near a playback deadline. To preserve GPU efficiency, an urgent request anchors each batch and compatible work fills the rest. Lower-level wins include trimming leading silence (about 80 ms off TTFA), ramping chunk sizes from small to large, capturing the Code Predictor’s fixed 15-step loop as a single CUDA graph, and a state-caching Codec that decodes only new frames instead of replaying the full utterance.

Both the implementation and the benchmark harness are open-sourced. The significance is less about a single model than about demonstrating that careful serving-layer engineering—not a bigger model or proprietary infrastructure—closes most of the gap with commercial TTS APIs, and does so on commodity single-GPU hardware with reproducible methodology.