Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit
4 days ago
- #KVCache Management
- #Hybrid SWA
- #Inference Optimization
- Hybrid Sliding Window Attention (Hybrid SWA) reduces KVCache storage and computation to about 1/7 of Full Attention, enhancing efficiency for long-context tasks.
- Sparse MoE activation maintains model capacity while reducing per-token compute, and multimodal encoders enable cross-modal understanding across vision, audio, and video.
- Optimizations include a dual KVCache pool for Full Attention and SWA layers, SWA-aware prefix cache trees for correct reuse, and a tiered caching system (GCache) for high performance and low cost.
- Scheduling strategies like KVCache affinity and priority for high-hit-rate requests improve throughput and reduce latency, especially in agentic scenarios.
- Memory optimizations, MTP support in prefill, and EPD disaggregation enhancements boost decode performance and encoder throughput.
- The implementation achieves high KVCache hit rates (averaging 93-95%), reduces expert parallelism size, and addresses load imbalance through length bucketing.