A brief history of KV cache compression developments

a day ago

KV cache compression has improved roughly 100x since 2017, mainly through algorithm innovations, while GPU memory only improved 18x.
The KV cache stores context for LLMs, making longer conversations more memory-efficient as compression improves.
Early KV caches using MHA required about 2.6MB of memory per token, making large contexts impractical on available hardware.
MQA in 2019 achieved 64x compression but suffered quality loss, while GQA in 2023 provided 8x compression with minimal quality degradation.
Sliding window attention and other methods further reduced cache growth, enabling rapid increases in context window length.
DeepSeek's MLA in 2024 compressed KV cache by 93% without quality loss, significantly reducing serving costs.
Quantization of KV cache to 8 or 4 bits effectively doubled or quadrupled capacity on top of other methods.
Linear-attention hybrids like Qwen3-Next and Kimi Linear enabled 1M token context windows by using fixed-size states.
Most efficiency gains were spent on longer context windows rather than lower costs, enabling more capable agents.
Future research aims to eliminate the quadratic attention bottleneck entirely, with potential for further compression breakthroughs.

Hasty Briefsbeta