inference optimization

Bilingual

Morph (YC S23) Is Hiring Member of Technical Stuff
7 hours ago
- Morph builds the inference infrastructure behind the fastest open models, spanning kernels, model serving, routing, autoscaling, and capacity.
- They are hiring a performance engineer to optimize the entire system, with a focus on PD disaggregation research and custom speculative-decoding models.
- The role involves working directly with founders on high-impact problems, using enormous compute in a small team, and includes a 2-day work trial.
Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit
a month ago
- Hybrid Sliding Window Attention (Hybrid SWA) reduces KVCache storage and computation to about 1/7 of Full Attention, enhancing efficiency for long-context tasks.
- Sparse MoE activation maintains model capacity while reducing per-token compute, and multimodal encoders enable cross-modal understanding across vision, audio, and video.
- Optimizations include a dual KVCache pool for Full Attention and SWA layers, SWA-aware prefix cache trees for correct reuse, and a tiered caching system (GCache) for high performance and low cost.
- Scheduling strategies like KVCache affinity and priority for high-hit-rate requests improve throughput and reduce latency, especially in agentic scenarios.
more...
Width vs. Depth: Speculating on the Margin
a month ago
- Running speculative decoding at batch size 1 with a 2-position draft can yield higher throughput than batching two sequences, even with a 10% token rejection rate.
- MoE routing causes co-activation of experts in speculative runs, reducing memory movement compared to random batching, making depth cheaper than width.
- Speculation efficiency varies per sequence; confidence gating allocates draft depth unevenly based on drafter confidence, improving throughput, especially at small batch sizes.
- Simulations show confidence gated speculation beats fixed-depth policies, but requires engine support for ragged speculation (different depths per sequence).
more...
Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction
a month ago
- Google researchers introduce a method to retrofit Multi-Token Prediction (MTP) onto frozen production models like Gemini Nano v3 to accelerate on-device inference without needing separate drafters.
- This approach addresses mobile constraints (energy, memory) by leveraging the main model's internal state via a lightweight MTP head, eliminating memory redundancy and reducing latency.
- MTP integration on Pixel 9 and 10 devices shows significant speedups (50%+) and energy savings in features like AI Notification Summaries and Proofread, while maintaining output compatibility.
- Future work includes exploring parallel decoding and branching techniques to handle language ambiguity more efficiently under strict mobile constraints.

inference optimization

Morph (YC S23) Is Hiring Member of Technical Stuff

Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

Width vs. Depth: Speculating on the Margin

Accelerating Gemini Nano Models on Pixel with Frozen Multi-Token Prediction