Sharing QKV Projections Cuts Transformer KV Cache by 50% With Minimal Quality Loss

A new arXiv paper challenges the assumption that attention requires three distinct query, key, and value projections. The authors systematically test three sharing constraints (Q-K=V, Q=K-V, and Q=K=V) across synthetic tasks, vision benchmarks, and language models up to 1.2B parameters trained on 10B tokens. The headline finding: tying keys and values into a single projection (Q-K=V) matches or slightly beats standard QKV transformers, while shrinking the KV cache by half at a cost of just 3.1% perplexity.

The sharing trick stacks with existing head-sharing schemes. Combining Q-K=V with GQA-4 yields 87.5% cache reduction, and pairing it with MQA reaches 96.9% — numbers that meaningfully change what is feasible for on-device inference. The authors argue Q-K=V works because keys and values naturally occupy similar representational subspaces and attention runs in a low-rank regime, whereas collapsing queries with keys destroys the directional asymmetry attention depends on.

The result reframes projection sharing as a form of weight tying that has been hiding in plain sight. For edge deployment, where KV cache size is often the binding memory constraint, the practical takeaway is concrete: you can drop a projection matrix without retraining your assumptions about model quality.