Ingesting 1Gbps of logs into ClickHouse for $180/month
7 hours ago
- #LogIngestion
- #CostOptimization
- #ClickHouse
- OpenData Buffer extensions enable 1.1 Gbps log transfer to ClickHouse for $180/month via S3.
- Achieved p50 latency of ~2.8s and p99 latency of ~10s with at-least-once delivery and deduplication.
- Used pipelined producer and consumer runtimes to maximize throughput by parallelizing S3 requests.
- S3 costs replaced Kafka, saving significantly compared to Apache Kafka ($24,667/mo) and WarpStream ($8,867/mo).
- Stable batch identities facilitate deduplication in ClickHouse, ensuring data consistency across retries.