Content-defined chunking added to Bazel
3 days ago
- #remote cache
- #content-defined chunking
- #build optimization
- BuildBuddy's Remote Cache uses Content-Defined Chunking (CDC) to transfer only changed bytes of large outputs, reducing data uploads and storage.
- CDC addresses the issue where small source changes ripple into large transitive actions (linking, bundling), causing unnecessary re-uploads and duplication.
- The implementation in Bazel shows 40% less data uploaded and 40% smaller disk cache, enabled with Bazel 8.7 or 9.1+ and --experimental_remote_cache_chunking flag.
- Transitive actions, like those combining dependencies via depset, often lead to new digests for outputs even with minimal changes, making CDC beneficial.
- CDC provides a generic solution without modifying individual tools, focusing on reusing byte chunks from cached versions of outputs.