- Workers AI serves large open models like Kimi K-series and GLM using GPU inference near users, but memory constraints make serving them challenging.
- Three optimizations are applied: quantizing the KV cache from BF16 to FP8, compressing model weights from FP8 to INT4 for GLM, and protecting the shared KV cache with integrity checks.
- KV cache quantization halves memory usage, allowing up to 1.37 million tokens for Kimi K2.6, and increases throughput by 41% at higher concurrency without accuracy loss.
- Weight compression reduces GLM's checkpoint from 705 GB to 421 GB, lowering per-GPU memory to 52 GB and improving decode speed by up to 55% at low concurrency, with negligible accuracy change.