Hasty Briefsbeta

Bilingual

Sub-second Postgres replication to ClickHouse from physical WAL

20 days ago
  • ClickHouse announced WalShadow, an open-source engine that replicates Postgres data directly from the physical WAL, eliminating the need for logical replication.
  • Benchmarks show commit-to-visible latency of about 200 ms and sustained throughput of 289K rows/sec, closely matching the source Postgres instance.
  • WalShadow consumes the same physical WAL stream used by Postgres replicas, decodes it outside the source, and writes ClickHouse-native blocks directly.
  • It supports the full replication lifecycle: initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers.
  • Complex schema changes like ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE are supported.
  • WalShadow is open source and available on GitHub; a private preview is launching for ClickHouse Managed Postgres.
  • The architecture uses four stages: tracking live schema via a shadow Postgres instance, parallel WAL decoding, building ClickHouse-native blocks, and parallel inserts.
  • Every row carries the source WAL position (_lsn), allowing out-of-order block arrivals while preserving correctness and latest-row-per-key semantics.
  • Compared with PeerDB, WalShadow achieved roughly 50x lower latency and more than 2x the sustained throughput in the benchmark.
  • The broader vision is a unified stack where Postgres handles transactions and ClickHouse handles analytics with minimal operational overhead.