Chorus: A fast WAL for object storage
8 hours ago
- #Object Storage
- #Low-Latency Writes
- #Write-Ahead Log
- Building data storage systems on object storage is popular but typically increases write latency; Rapid Storage Buckets offer appendable objects with low latency (2 ms P99 commit latency) but are zonal.
- Chorus is a single-writer write-ahead log on Rapid Storage Buckets, using a regional metadata register for CAS operations and replicating appends across zones, achieving fast writes with zone-fault tolerance (P99 faster than Hyperdisk HA's P50).
- Chorus features formal modeling (using P), a high-performance Rust client, deterministic simulation testing, and open-source code; its protocol manages metadata (epoch, owner, segments) to ensure consistency and safe crash recovery without LIST APIs.
- The system supports segment rotation and prefix truncation, with pending and active segments to avoid latency spikes during rotation; recovery handles inconsistencies by copying consistent objects.
- Chorus aims to enable robust, bespoke data storage systems on GCP, potentially used with SlateDB, reflecting a trend towards specialized storage systems due to improved primitives.