30x faster than Prometheus: How we rebuilt ES as a leading columnar datastore
a day ago
- #Elasticsearch
- #Time Series
- #Performance Optimization
- Elasticsearch now stores OpenTelemetry metrics at 3.75 bytes per data point, down from 25 bytes a year ago, improving storage efficiency by 2.5x compared to Prometheus, Mimir, and ClickHouse.
- Performance gains include up to 30x faster query speeds and up to 50% higher indexing throughput due to a columnar metrics engine built from TSDS and ES|QL.
- Storage optimizations in versions 9.1–9.4 reduced footprint via doc value skippers (-10 bytes), synthetic IDs (-5 bytes), trimming sequence numbers (-4 bytes), and larger codec blocks (-2 bytes).
- Query execution benefits from vectorized processing, zero-copy decoding, and optimized counter rate evaluation, with ES|QL's TS source command enabling efficient time series aggregations.
- Elasticsearch supports OTLP protobuf and Prometheus remote write entrypoints, reducing parsing overhead and improving indexing throughput by up to 20%.
- Benchmarks show Elasticsearch outperforms competitors by up to 30x in gauge average and counter rate queries, and up to 5x in prefix filter queries.
- Prometheus integration (tech preview in v9.4) allows querying with PromQL and ES|QL, enabling migration without dashboard changes.