Speculative KV Coding: Lossless 4x Compression of LLM Cache via Predictor Models

As LLM contexts balloon and agentic workflows push memory demands higher, the KV cache that makes long contexts affordable is itself becoming the bottleneck. Lossy approaches like TurboQuant shrink the cache by dropping bit-width, but the quality cost is only discoverable after the fact through evaluations. Fergus Finn proposes a lossless alternative: Speculative KV coding, which compresses the cache of a large target model by roughly 4x using a cheaper predictor model whose forward pass on the same prompt approximates what the target’s cache will be.

The technique borrows from speculative decoding. Because a KV cache is deterministic output rather than truly random data, the bitrate an arithmetic coder spends is pure KL divergence between the predictor and the truth. Modeling the per-scalar error as a Gaussian centered on the predictor’s output, with variance calibrated to typical residual size, yields an expected bitrate equal to the log of the typical error magnitude. Both encoder and decoder rerun the same predictor on the prompt to reconstruct identical parameters; only the encoder touches the target model and emits a bitstream that the decoder uses to recover the exact cache.

The most practical predictor turns out to be an optimized variant of the target itself, such as an FP8 quantized version that ships alongside the full-precision weights. Residual variance is fit once on calibration data and frozen, so deployment requires no training. Early results show meaningful gains on top of existing lossy FP8 compression, suggesting a path toward shrinking cache footprints without sacrificing fidelity.