Kapa cuts RAG context 68% with a cheap LLM pruner, keeping 96% recall

Kapa, which builds AI assistants over large product knowledge bases, inserted a third stage into its RAG pipeline to attack a specific cost problem: the generator model is billed for every retrieved chunk it reads, even the ones it ignores. Retrievers deliberately over-fetch for recall and trust the large model to filter noise, but in Kapa’s assistants those retrieved chunks account for roughly two-thirds of a query’s cost. Their fix is a small, fast LLM that sits between the reranker and the generator, reads the question alongside all retrieved chunks, and discards the ones the answer won’t need before the expensive model ever sees them.

The key insight is that relevance is a property of a set, not of an individual chunk. Simpler approaches failed for this reason: rerank scores are only relative orderings and can’t be cut with a fixed threshold, and pointwise cross-encoders judge each chunk in isolation, so they miss chunks that are only meaningful next to another one or that each answer part of a multi-part question. Even a clever anchor-document trick that calibrates the reranker couldn’t fix the underlying scoring problem. What worked was a single listwise LLM call that grades every chunk against a five-level, word-defined scale, so a fixed cutoff finally holds and partial or indirect relevance has somewhere to land. A keep-top-k rule protects the strongest chunks from grading mistakes.

Evaluated on labeled questions and replayed over a month of real production traffic, the relevance-scoring approach dominated both naive top-N truncation and simpler LLM designs at every compression level. Kapa shipped an aggressive configuration: about 68% of chunks dropped, roughly 96% of recall preserved (one question in 25 loses a needed chunk), and a net 34% reduction in per-query cost after accounting for the pruner itself. The tradeoff is latency, since the pruner sits in the critical path, but running a small model at low reasoning effort keeps the added time to about 0.7 seconds per query.