- Workers AI serves large open models like Kimi K-series and GLM using GPU inference near users, but memory constraints make serving them challenging.
- Three optimizations are applied: quantizing the KV cache from BF16 to FP8, compressing model weights from FP8 to INT4 for GLM, and protecting the shared KV cache with integrity checks.
- KV cache quantization halves memory usage, allowing up to 1.37 million tokens for Kimi K2.6, and increases throughput by 41% at higher concurrency without accuracy loss.
- Weight compression reduces GLM's checkpoint from 705 GB to 421 GB, lowering per-GPU memory to 52 GB and improving decode speed by up to 55% at low concurrency, with negligible accuracy change.
- Integrity checks on the KV cache add less than 1% overhead on throughput and latency, preventing data corruption from shared page reallocation at high request volumes.
- These optimizations enable more customers to be served at lower cost while maintaining model accuracy, with future work on FP8 caches, NVFP4 weights, and further safety checks.