Huawei's KVarN squeezes 3-5x more KV-cache into vLLM without losing FP16 accuracy

Huawei’s CSL group released KVarN, a vLLM attention backend that quantizes the KV cache aggressively while keeping FP16-level accuracy and throughput. On Qwen3-32B benchmarks, it delivers roughly 4x the KV-cache capacity, matches FP16 on AIME25 reasoning tasks, and beats FP16 throughput — a combination that prior approaches like TurboQuant couldn’t reach, since they typically traded 40-52% of throughput for capacity gains.

The technique works tile-by-tile: a Hadamard rotation smears per-channel outliers, an iterative Sinkhorn-like normalization equalizes variance across the tile, and asymmetric round-to-nearest does the actual low-bit quantization. The shipped preset uses 4-bit keys and 2-bit values, which the authors picked as the highest-fidelity configuration. It’s calibration-free and ships as a vLLM fork — users add a kv_cache_dtype flag, no model changes required.

The pitch targets agentic and long-context serving, where KV-cache memory is the binding constraint on concurrency and context length. Released under Apache 2.0 and built on vLLM 0.22.0, with kernels in Triton and JIT-compiled at runtime. Tile size is currently fixed at 128 tokens, with other page sizes promised later.