Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context

14 days ago

Laguna XS 2.1 33B model achieves 296 tok/s peak and 152 tok/s at 256K context on a single RTX 3090 GPU using speculative decoding and optimizations.
Key optimizations include a drafter KV ring cache, sliding-window ring caches, and KVFlash paging, making decode speed independent of context length.
Prefill performance improved from 411 seconds to 67 seconds for 256K tokens, with remaining costs mainly from MoE expert GEMMs.
Speculative decoding is lossless, ensuring output quality matches the model's own, as confirmed by benchmarks like GSM8K 10/10.
The optimizations are model-agnostic and integrated into the lucebox engine, enabling efficient long-context processing on limited GPU memory.

Hasty Briefsbeta