- KV cache compression has improved roughly 100x since 2017, mainly through algorithm innovations, while GPU memory only improved 18x.
- The KV cache stores context for LLMs, making longer conversations more memory-efficient as compression improves.
- Early KV caches using MHA required about 2.6MB of memory per token, making large contexts impractical on available hardware.
- MQA in 2019 achieved 64x compression but suffered quality loss, while GQA in 2023 provided 8x compression with minimal quality degradation.
- Sliding window attention and other methods further reduced cache growth, enabling rapid increases in context window length.
- DeepSeek's MLA in 2024 compressed KV cache by 93% without quality loss, significantly reducing serving costs.
- Quantization of KV cache to 8 or 4 bits effectively doubled or quadrupled capacity on top of other methods.
- Linear-attention hybrids like Qwen3-Next and Kimi Linear enabled 1M token context windows by using fixed-size states.
- Most efficiency gains were spent on longer context windows rather than lower costs, enabling more capable agents.
- Future research aims to eliminate the quadratic attention bottleneck entirely, with potential for further compression breakthroughs.