A Fork in the Road: Deciding Kafka's Diskless Future
6 months ago
- #Kafka
- #Cloud-Computing
- #Open-Source
- The Kafka community is addressing high replication costs across cloud availability zones with three KIPs (KIP-1150, KIP-1176, KIP-1183).
- Kafka faces a pivotal decision on implementing S3 topics, impacting its long-term success and architectural direction.
- Two opposing forces shape Kafka's future: stabilizing on-prem deployments and disrupting cloud-native workloads.
- Three KIPs propose different approaches to integrate object storage into Kafka, focusing on cost savings and architectural benefits.
- KIP-1150 (revolutionary) aims for a direct-to-S3 design, maximizing elasticity and operational simplicity but with higher implementation costs.
- KIP-1176 (evolutionary) seeks to reuse existing Kafka components, focusing on cost savings but potentially limiting architectural benefits.
- KIP-1183 is considered non-viable due to lack of implementation and community benefit.
- Key challenges include sequencing, metadata storage, and ensuring correctness in leaderless designs.
- WarpStream and Aiven Inkless serve as reference implementations for revolutionary designs, leveraging stateless compute and object storage.
- The community must decide between revolutionary and evolutionary paths, balancing innovation with maintainability.