Improving storage efficiency in Magic Pocket, Dropbox's immutable blob store

6 days ago

Magic Pocket is Dropbox's core custom blob storage system, designed for durability, availability, and efficiency at exabyte scale.
A new service for data placement reduced write amplification but inadvertently increased fragmentation, leading to higher storage overhead due to under-filled volumes.
Immutable blob storage means blobs are never modified in place; deletes don't free space immediately, requiring compaction to reclaim it.
Erasure coding is used for redundancy, providing fault tolerance with less storage than replication, but fragmentation affects how efficiently space is used.
An issue with the Live Coder service caused severely under-filled volumes, increasing overhead and prompting a rethink of compaction strategies.
The existing L1 compaction strategy was effective in steady state but couldn't handle the long tail of under-filled volumes efficiently.
L2 compaction was introduced to consolidate moderately under-filled volumes using dynamic programming for faster space reclamation.
L3 compaction targets the sparsest volumes by streaming live blobs through Live Coder, enabling quick reclamation with tradeoffs like metadata updates.
Operational safeguards include dynamic tuning of eligibility thresholds, rate limiting, and traffic locality to prevent system overload.
Key lessons include the need for multiple compaction strategies, automated tuning, monitoring metadata capacity, and improved visibility for proactive management.

Hasty Briefsbeta