It was surprisingly hard to break CloudNativePG replication
a day ago
- The author tried to create a Postgres replica that lags behind its primary but found it kept catching up because the replica was pulling WAL from the S3 archive when streaming was cut.
- The replica's replay LSN advanced even without streaming, thanks to CloudNativePG's restore_command that fetches WAL from object storage.
- Postgres's standby recovery state machine falls back to archive when streaming fails, creating a loop that keeps the replica nearly caught up.
- CloudNativePG configures both primary_conninfo and restore_command, enabling the fallback; its wal-restore binary prefetches segments from S3.
- To actually break replication, both streaming and archive access must be blocked; the author used Chaos Mesh to cut the replica off from the S3 endpoint.
- Breaking things on purpose is a valuable way to understand system behavior and to ensure monitoring tools produce meaningful alerts.