- Kimi Linear is a hybrid linear attention architecture that outperforms full attention in short-context, long-context, and reinforcement learning scenarios.
- The core innovation is Kimi Delta Attention (KDA), an expressive linear attention module with a finer-grained gating mechanism that improves finite-state RNN memory usage.
- A specialized chunkwise algorithm using Diagonal-Plus-Low-Rank transition matrices reduces computation while maintaining consistency with the classical delta rule.
- The 3B activated/48B total parameter model, hybridizing KDA and Multi-Head Latent Attention, outperforms full MLA across tasks with up to 75% less KV cache and 6x decoding throughput for 1M context.
- Kimi Linear serves as a drop-in replacement for full attention architectures with superior performance and efficiency.
- Open-source contributions include the KDA kernel, vLLM implementations, and pre-trained/instruction-tuned model checkpoints.