Divide and Compact: Segment-Oriented Compaction in SlateDB
14 hours ago
- #LSM trees
- #database optimization
- #compaction strategies
- LSMs use a single tree for data, causing challenges in managing read/write amplification for mixed data structures with different patterns or lifetimes.
- Segment-oriented compaction in SlateDB splits the dataset into separate trees, allowing tailored compaction strategies based on data structure, frequency, retention, or other dimensions.
- Compaction in SlateDB involves L0 tables and sorted runs, with a scheduler like size-tiered compaction selecting similarly sized runs, though it struggles with heterogeneous data or time-based structures like timeseries.
- Segment-oriented compaction, introduced in RFC 24, partitions the keyspace by prefix, creating isolated LSM trees per segment for customized compaction and retention, useful for index structures or timeseries data.
- Benefits include bounded compaction work, steady throughput, improved write performance, and efficient retention, but tradeoffs include potential L0 write costs, manifest overhead, and the need for proper segment sizing for query efficiency.