replication

Bilingual

OrioleDB beta15 and beta16: stability, testing, and replication hardening
2 months ago
- Beta15 and beta16 focus exclusively on stability, correctness, and test coverage for existing features, especially replication, recovery, and concurrent workloads.
- A new testing approach runs PostgreSQL's own regression and isolation tests on OrioleDB tables, catching compatibility issues quickly.
- Automated primary/replica data comparison in CI verifies replication correctness, catching bugs invisible on single nodes.
- Crash consistency testing using dm-log-writes simulates crashes at every recovery point, exercising WAL durability and checkpoint atomicity.
- CI infrastructure expanded to cover PostgreSQL 16/17/18, nightly builds, injection points, TAP logs, and valgrind integration.
- Recovery and replication hardening includes fixes for parallel recovery, undo management, bridge indexes, and logical replication.
- New functionality: SERIALIZABLE isolation support and backend-local page pool for temporary tables to reduce shared buffer contention.
- SQL compatibility improvements include hashing, TOAST handling, generated columns, domain types, and scan key fixes.
It was surprisingly hard to break CloudNativePG replication
2 months ago
- The author tried to create a Postgres replica that lags behind its primary but found it kept catching up because the replica was pulling WAL from the S3 archive when streaming was cut.
- The replica's replay LSN advanced even without streaming, thanks to CloudNativePG's restore_command that fetches WAL from object storage.
- Postgres's standby recovery state machine falls back to archive when streaming fails, creating a loop that keeps the replica nearly caught up.
- CloudNativePG configures both primary_conninfo and restore_command, enabling the fallback; its wal-restore binary prefetches segments from S3.
- To actually break replication, both streaming and archive access must be blocked; the author used Chaos Mesh to cut the replica off from the S3 endpoint.
- Breaking things on purpose is a valuable way to understand system behavior and to ensure monitoring tools produce meaningful alerts.
<antirez>
2 months ago
- During the first Redis developers meeting, diskless replication was identified as a high-priority feature to avoid disk usage during slave synchronization.
- In traditional replication, full resynchronization requires creating an RDB file on disk and sending it to slaves, causing disk pressure and latency issues.
- Diskless replication allows the master's child process to write RDB data directly to slave sockets without intermediate disk writes, addressing use cases with slow disks or caching environments.
- The implementation uses a rio abstraction to write to file descriptors, a random EOF marker to indicate transfer completion, and SO_SNDTIMEO for timeout handling.
- Multiple slaves can be served simultaneously by waiting a few seconds for new slaves and writing data in small chunks to each file descriptor in a loop.
- Partial failures are managed by marking failed FDs with error codes and using a pipe to report slave transfer status to the parent process.
- Diskless replication enables a fully disk-free Redis setup, leading to plans for improved support of replication without persistence.
<antirez>
2 months ago
- A company experienced data loss when a Redis master with disabled persistence was restarted, causing a split-brain and replication of an empty dataset to all slaves.
- Redis Sentinel does not protect against misconfiguration like disabling persistence; this is a byzantine failure.
- Current Sentinel slave selection uses replication offset, which resets on restart and does not verify the master's runid, leading to potential promotion of outdated slaves.
- Proposed improvement: tie replication offset to the master's runid to ensure slaves have continuity with the failed master.
- Better Sentinel documentation is needed, including a list of common misconfigurations to avoid, such as turning off persistence.

replication

OrioleDB beta15 and beta16: stability, testing, and replication hardening

It was surprisingly hard to break CloudNativePG replication

<antirez>

<antirez>