Show HN: Walrus – a Kafka alternative written in Rust
10 days ago
- #raft-consensus
- #message-streaming
- #distributed-systems
- Walrus is a distributed message streaming platform with high-performance log storage.
- Key features include automatic load balancing, fault tolerance via Raft consensus, and high-performance storage with io_uring on Linux.
- Producers and consumers can connect to any node, with automatic request routing to segment leaders.
- Each node consists of Node Controller, Raft Engine, Cluster Metadata, and Bucket Storage components.
- The system uses a simple length-prefixed text protocol over TCP for client interactions.
- Topics are split into segments, each with a leader node for writes, and leadership rotates on segment rollover.
- Leases derived from Raft-replicated metadata ensure write consistency and prevent split-brain scenarios.
- Sealed segments allow historical reads from any replica without data movement.
- Comprehensive test suite includes tests for basic operations, rollover, resilience, recovery, stress, and multi-topic scenarios.
- Performance benchmarks show Walrus outperforming Kafka and RocksDB in throughput and bandwidth.
- Includes a formal TLA+ specification for modeling distributed data plane correctness.
- Core Walrus storage engine is available as a standalone Rust library for embedded use cases.
- Project is open-source under MIT License, with contributions welcome.