speculative decoding

Bilingual

Kimi K3 on vLLM: Up to 370 Tokens/sec
2 months ago
- Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 16 of 896 experts active per token, a 1M-token context window, and native vision, built on Kimi Delta Attention (KDA) and Attention Residuals.
- vLLM provides efficient day-0 support for Kimi K3, adapting its architecture with a hybrid KV-cache manager that handles both KDA recurrent state and full-attention paged KV under one scheduler.
- The model achieves up to 370 tok/s per user with DSpark speculative decoding (3.14× improvement over 118 tok/s without speculation) on 16 NVIDIA GB300 NVL72 GPUs.
- Production features supported include speculative decoding (DSpark), prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, structured output, and multi-GPU support (NVIDIA Hopper/Blackwell, AMD MI355X).
- vLLM redesigned prefix caching for KDA's recurrent state using hybrid interval-based and Marconi-style selective retention to optimize cache space and reuse.
- Performance optimizations include fused KDA decode kernels, low-latency BF16 GEMM (skinnyGEMM), MoE tail fusion, and sequence parallelism for prefill to reduce communication overhead.
- Accuracy evaluations show strong scores: GSM8K 0.976, GPQA-Diamond 0.939, OCRBench 0.889, MMMU Pro Vision 0.818 at maximum reasoning effort.
- Deployment tips: enable prefix caching explicitly, use appropriate MoE/all-to-all backends (deep_gemm_mega_moe for DEP, flashinfer_trtllm for TP), and set Rust frontend for full support.
Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context
2 months ago
- Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
- Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
- Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.
- Speculative decoding is lossless, ensuring output quality matches the model's own, as confirmed by benchmarks like GSM8K 10/10.
- The optimizations are model-agnostic and integrated into the lucebox engine, enabling efficient long-context processing on limited GPU memory.
Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s
2 months ago
- Achieved a record of 511.6 tokens per second serving Kimi-K2.6, a 1 trillion-parameter model, on four B200 GPUs with lossless speculative decoding.
- The record was set at 505.9 tokens per second and raised via blind re-runs using publicly available components, outperforming other providers on public leaderboards by up to 13.4%.
- Measured the 'verify tax' at 0.397ms per token in trillion-parameter MoE models, showing that verification cost scales with draft length due to expert routing.
- Developed Fovea, a trained drafter head that improves performance by 5.8% over public heads, demonstrating the potential of domain-specific optimization.
- Identified and fixed a training defect where distillation teacher was misaligned by one transformer layer, highlighting the importance of correct hidden-state capture in drafter training.
Width vs. Depth: Speculating on the Margin
3 months ago
- Running speculative decoding at batch size 1 with a 2-position draft can yield higher throughput than batching two sequences, even with a 10% token rejection rate.
- MoE routing causes co-activation of experts in speculative runs, reducing memory movement compared to random batching, making depth cheaper than width.
- Speculation efficiency varies per sequence; confidence gating allocates draft depth unevenly based on drafter confidence, improving throughput, especially at small batch sizes.
- Simulations show confidence gated speculation beats fixed-depth policies, but requires engine support for ragged speculation (different depths per sequence).
- Speculative decoding remains critical for inference optimization as models hit memory walls, with ongoing rapid evolution of techniques.

speculative decoding

Kimi K3 on vLLM: Up to 370 Tokens/sec

Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context

Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s

Width vs. Depth: Speculating on the Margin