Data liberation: Apache Kafka's native cluster mirroring
2 days ago
- Cluster mirroring embeds cross-cluster replication directly into the Kafka broker, eliminating external tools like MirrorMaker 2 and reducing operational complexity.
- It replicates data byte-for-byte without decompression or recompression, preserving exact offsets across clusters for seamless failover.
- The architecture includes MirrorMetadataManager (orchestrator), ClusterMirrorCoordinator (state persistence), and MirrorFetcherThread (data fetching using standard fetch protocol).
- Mirror partition lifecycle follows a state machine with stages: LOG_ALIGNMENT, EPOCH_FENCING, MIRRORING, ULE_RECOVERY, PAUSING/PAUSED, STOPPING/STOPPED, and FAILED.
- Data consistency is maintained via log convergence (epoch alignment and truncation) and transaction safety (ABORT markers for in-flight transactions and PID reset during stop).
- Disaster recovery uses one-command failover and failback, with LME records enabling incremental re-replication and preserving consumer group offsets.
- Cluster migration supports sources as old as Kafka 2.1, allowing a clean upgrade to modern KRaft clusters without intermediate version upgrades.