Zstandard Across the Stack
11 hours ago
- #data compression
- #game development
- #bandwidth optimization
- Zstandard (zstd) with custom dictionaries is used to compress real-time game messages, reducing bandwidth by 70-90% with minimal CPU overhead.
- Custom dictionaries are built from training data and embedded in clients and servers, allowing gRPC to negotiate compression via headers for seamless upgrades.
- The same dictionary approach is applied to database writes and reads to reduce I/O, and for patching game updates, enabling players to download only changed data instead of full redownloads.
- For full downloads, zstd with streaming decompression replaces .zip, allowing overlapping download and extraction phases and improving compression by 13% at level 19.
- The flexibility of zstd's dictionaries, streaming, and tunable levels addresses multiple stack problems, showing compression's broader applications beyond file archiving.