The negentropy set-reconciliation protocol
a day ago
- #data-synchronization
- #protocol
- #distributed-systems
- NeGentropy is a set-reconciliation protocol for syncing datasets efficiently.
- It helps detect and fix inconsistencies in data sets due to network issues or independent creation.
- The protocol involves two parties (client and server) exchanging messages to identify missing records.
- Records must be mapped to a 32-byte ID (typically a cryptographic hash) and a timestamp.
- NeGentropy does not support updating records; changes must be modeled as deletions and insertions.
- The protocol uses three message modes: Skip, Fingerprint, and IdList for processing ranges.
- Fingerprint ranges use incremental hashing for efficiency in detecting differences.
- Implementations may support frame size limits to manage message sizes and network constraints.
- Multiple language implementations exist, including C++, JavaScript, Rust, Go, and Kotlin.
- A conformance test suite is available to ensure interoperability between implementations.