2 days ago
- Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
- vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
- The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.
- Production features supported include speculative decoding (DSpark), prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, structured output, and multi-GPU support (NVIDIA Hopper/Blackwell, AMD MI355X).
- vLLM redesigned prefix caching for KDA's recurrent state using hybrid interval-based and Marconi-style selective retention to optimize cache space and reuse.
- Performance optimizations include fused KDA decode kernels, low-latency BF16 GEMM (skinnyGEMM), MoE tail fusion, and sequence parallelism for prefill to reduce communication overhead.
- Accuracy evaluations show strong scores: GSM8K 0.976, GPQA-Diamond 0.939, OCRBench 0.889, MMMU Pro Vision 0.818 at maximum reasoning effort.
- Deployment tips: enable prefix caching explicitly, use appropriate MoE/all-to-all backends (deep_gemm_mega_moe for DEP, flashinfer_trtllm for TP), and set Rust frontend for full support.