Bilingual

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction
25 days ago
- Google researchers introduce a method to retrofit Multi-Token Prediction (MTP) onto frozen production models like Gemini Nano v3 to accelerate on-device inference without needing separate drafters.
- This approach addresses mobile constraints (energy, memory) by leveraging the main model's internal state via a lightweight MTP head, eliminating memory redundancy and reducing latency.
- MTP integration on Pixel 9 and 10 devices shows significant speedups (50%+) and energy savings in features like AI Notification Summaries and Proofread, while maintaining output compatibility.
- Future work includes exploring parallel decoding and branching techniques to handle language ambiguity more efficiently under strict mobile constraints.
Width vs. Depth: Speculating on the Margin
25 days ago
- Running speculative decoding at batch size 1 with a 2-position draft can yield higher throughput than batching two sequences, even with a 10% token rejection rate.
- MoE routing causes co-activation of experts in speculative runs, reducing memory movement compared to random batching, making depth cheaper than width.
- Speculation efficiency varies per sequence; confidence gating allocates draft depth unevenly based on drafter confidence, improving throughput, especially at small batch sizes.
- Simulations show confidence gated speculation beats fixed-depth policies, but requires engine support for ragged speculation (different depths per sequence).
- Speculative decoding remains critical for inference optimization as models hit memory walls, with ongoing rapid evolution of techniques.
Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit
20 days ago
- Hybrid Sliding Window Attention (Hybrid SWA) reduces KVCache storage and computation to about 1/7 of Full Attention, enhancing efficiency for long-context tasks.
- Sparse MoE activation maintains model capacity while reducing per-token compute, and multimodal encoders enable cross-modal understanding across vision, audio, and video.
- Optimizations include a dual KVCache pool for Full Attention and SWA layers, SWA-aware prefix cache trees for correct reuse, and a tiered caching system (GCache) for high performance and low cost.
- Scheduling strategies like KVCache affinity and priority for high-hit-rate requests improve throughput and reduce latency, especially in agentic scenarios.
- Memory optimizations, MTP support in prefill, and EPD disaggregation enhancements boost decode performance and encoder throughput.
- The implementation achieves high KVCache hit rates (averaging 93-95%), reduces expert parallelism size, and addresses load imbalance through length bucketing.

inference optimization

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction

Width vs. Depth: Speculating on the Margin

Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit