Saving another 100TB of RAM with math (and Rust)
4 hours ago
- Cloudflare reduced memory usage of a Pingora-based service by over 100TB globally through algorithm and data structure improvements.
- Consistent hashing distributes tasks across servers using hash outputs on a number line, but can lead to workload imbalance without multiple hashes per server.
- Adding multiple hashes per server improves workload distribution, with 160 hashes per server reducing coefficient of variation from 99% to 8%.
- Memory was saved by compacting a Point struct from 8 bytes to 6 bytes using byte arrays, reducing memory by 25%.
- Mathematical derivation showed fewer hashes per server (90% reduction) could be used without significant error, enabling further memory savings.
- Migration to the new smaller hash ring was done gradually per data center to avoid cache invalidation and origin traffic spikes.
- The changes are available in the pingora-ketama crate with a v2 ring feature for compact storage and scalable hashes.