- Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
- Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
- Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.
- Speculative decoding is lossless, ensuring output quality matches the model's own, as confirmed by benchmarks like GSM8K 10/10.