a month ago
- Pipelined decoding overlaps GPU and CPU work to avoid GPU bubbles, where the GPU waits for CPU housekeeping.
- Three mechanisms ensure safety: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie refcounting manages finished requests.
- Performance gains depend on GPU speed and batch size, with up to 35% improvement on faster hardware as bubbles become more significant.