- The article derives Kimi Delta Attention (KDA) from softmax attention through a series of improvements: linear attention, DeltaNet, Gated DeltaNet, and finally KDA.
- Linear attention removes softmax and uses a fixed-size recurrent state of outer products, enabling O(n) time but losing selectivity and causing interference between writes.
- DeltaNet corrects interference by writing the prediction error (delta rule) instead of the full value, making the write targeted and local in key space.
- Gated DeltaNet adds a scalar retention gate for global forgetting before applying the delta rule, but forgets all key channels at the same rate.
- KDA replaces the scalar gate with a per-key-channel vector gate, resulting in a diagonal-plus-low-rank (DPLR) state transition that allows independent forgetting per key channel.
- The recurrent Triton kernel implements KDA token-by-token for efficient autoregressive decode, using vector operations and outer products.
- The chunkwise implementation processes tokens in groups using matrix products, enabling tensor-core-efficient training and prefill via causal triangular solves and WY-style factors.
- Both the recurrent and chunkwise programs are two schedules of the same KDA recurrence, not different attention mechanisms.