An adventure in writing compatible systems
13 days ago
- #Rust
- #Database
- #SQLite
- Turso is a Rust rewrite of SQLite aiming for full compatibility while adding new features like CDC, concurrent writes, and encryption.
- A surprising compatibility issue was found where inserting more than 1GB of data caused SQLite integrity checks to fail.
- Deterministic Simulation Testing (DST) is heavily used but initially didn't catch the issue because test databases didn't reach 1GB due to fault injection.
- After improving the simulator to allow databases to grow beyond 1GB, the corruption issue appeared consistently.
- Investigation revealed that SQLite inserts a special page at the 1GB mark, which Turso wasn't handling, leading to corruption.
- Nikita Sivukhin, a highly productive team member, identified the issue by knowing about the special page from SQLite's source code.
- The fix was implemented once the undocumented behavior was understood, highlighting the challenges of rewriting complex systems.