When rust ≠ performance. a lesson in developer experience
6 hours ago
- #rust
- #rocksdb
- #performance
- Oxen aims to be the fastest data versioning tool, requiring multi-terabyte benchmarks for commands like add and commit.
- The commit command was significantly slower than add, taking >50 minutes for 1M files despite being an O(n) operation.
- Profiling revealed >90% of commit time was spent acquiring a lock on the staging RocksDB due to thread contention.
- Excessive .clone() and db.open() operations across layers caused redundant data fetching and performance bottlenecks.
- A PR reduced commit time by 20x by minimizing redundant operations and optimizing system design.
- RocksDB, optimized for parallel writes, was suboptimal for Oxen's parallel reads use case, leading to unexpected overhead.
- The lesson: simplicity and holistic system design are crucial, even for isolated features.