Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context
9 hours ago
- #Speculative Decoding
- #AI Inference Optimization
- #GPU Performance
- Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
- Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
- Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.
- Speculative decoding is lossless, ensuring output quality matches the model's own, as confirmed by benchmarks like GSM8K 10/10.
- The optimizations are model-agnostic and integrated into the lucebox engine, enabling efficient long-context processing on limited GPU memory.