Bilingual

Width vs. Depth: Speculating on the Margin
25 days ago
- Running speculative decoding at batch size 1 with a 2-position draft can yield higher throughput than batching two sequences, even with a 10% token rejection rate.
- MoE routing causes co-activation of experts in speculative runs, reducing memory movement compared to random batching, making depth cheaper than width.
- Speculation efficiency varies per sequence; confidence gating allocates draft depth unevenly based on drafter confidence, improving throughput, especially at small batch sizes.
- Simulations show confidence gated speculation beats fixed-depth policies, but requires engine support for ragged speculation (different depths per sequence).
- Speculative decoding remains critical for inference optimization as models hit memory walls, with ongoing rapid evolution of techniques.
Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s
21 days ago
- Achieved a record of 511.6 tokens per second serving Kimi-K2.6, a 1 trillion-parameter model, on four B200 GPUs with lossless speculative decoding.
- The record was set at 505.9 tokens per second and raised via blind re-runs using publicly available components, outperforming other providers on public leaderboards by up to 13.4%.
- Measured the 'verify tax' at 0.397ms per token in trillion-parameter MoE models, showing that verification cost scales with draft length due to expert routing.
- Developed Fovea, a trained drafter head that improves performance by 5.8% over public heads, demonstrating the potential of domain-specific optimization.
- Identified and fixed a training defect where distillation teacher was misaligned by one transformer layer, highlighting the importance of correct hidden-state capture in drafter training.
Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context
16 days ago
- Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
- Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
- Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.
- Speculative decoding is lossless, ensuring output quality matches the model's own, as confirmed by benchmarks like GSM8K 10/10.
- The optimizations are model-agnostic and integrated into the lucebox engine, enabling efficient long-context processing on limited GPU memory.

speculative decoding

Width vs. Depth: Speculating on the Margin

Serving a 1T open model at 511 tok.s single-stream, lossless, on four B200s

Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context