rqlite 9.0: Real-Time Change Data Capture for Distributed SQLite
6 hours ago
- #database
- #CDC
- #distributed-systems
- rqlite is a lightweight, open-source, distributed relational database written in Go, using SQLite as its storage engine and Raft for consensus.
- rqlite 9.0 introduces Change Data Capture (CDC), enabling streaming of database changes to external systems, turning rqlite into a live event source.
- CDC captures INSERTs, UPDATEs, and DELETEs, pushing details (table, rows, old/new values) to a user-defined HTTP endpoint, eliminating the need for polling.
- Enabling CDC is straightforward by specifying a webhook endpoint URL; rqlite POSTs change events to this URL as they occur.
- CDC events are packaged in JSON, containing metadata, Raft index, commit timestamp, and details of the change (operation type, table, row data).
- Design challenges included reliably capturing changes, ensuring at-least-once delivery, handling node restarts/leadership changes, and managing event queues.
- rqlite uses a disk-backed FIFO queue for CDC events, with a high-water mark mechanism to minimize duplicates and ensure event delivery.
- Events are batched and retried if delivery fails, with configurable retry policies to handle slow or unreachable endpoints.
- Future improvements include streaming to systems like Apache Kafka and emitting events for schema changes (e.g., CREATE TABLE).