More databases should be single-threaded
18 hours ago
- #database
- #sharding
- #scalability
- Traditional SQL databases face issues with transaction modes (READ COMMITTED, REPEATABLE READ, SERIALIZABLE) leading to deadlocks, race conditions, and scaling bottlenecks.
- Single-writer databases like SQLite avoid conflicting writes by ensuring only one writer at a time, but they don't scale beyond a single writer.
- Sharding databases can help with scaling but introduces challenges like cross-shard queries, complex infrastructure, and difficult transactions/migrations.
- Optimistic concurrency, sagas, and two-phase commits can mitigate some sharding issues, but they come with trade-offs.
- Sharded single-thread databases offer conceptual purity, serializable transactions, and horizontal scalability, but require early sharding commitment.
- Existing solutions like ScyllaDB and VoltDB use shard-per-core concepts but target high-scale applications, leaving a gap for average web developers.
- A 'mostly Postgres-compatible' database with serializable transactions and good sharding support could solve many common problems without requiring a full paradigm shift.