Pruning RAG context down to what the answer actually needs
7 hours ago
- #context pruning
- #RAG optimization
- #LLM efficiency
- A small, cheap LLM was introduced between retrieval and generation to prune irrelevant context chunks.
- The pruner drops about 68% of retrieved chunks while maintaining 96% recall, reducing query costs by approximately 34%.
- Relevance grading uses a five-level scale (ESSENTIAL to UNRELATED), judged with the full context to account for partial or indirect relevance.
- Pruning adds about 0.7 seconds of latency per query but is marginal in agent workflows with multiple model calls.
- The technique addresses limitations of pointwise rerankers by evaluating chunk sets collectively, improving compression-recall trade-offs.