Lies, Damn Lies and Database Benchmarks
2 days ago
- #database benchmarks
- #ClickBench
- #performance evaluation
- Benchmarks, like ClickBench, can be misleading due to subtle methodological nuances.
- Two key tweaks can significantly alter results: keeping processes alive for cache warmth and increasing iteration counts for JIT-based engines.
- In one scenario, DuckDB moved from third to first place when its process was kept resident, showing the impact of buffer pool warmth.
- In another scenario, JVM-based engines (like QuestDB and CrateDB) improved with more iterations due to JIT optimization warmup.
- Rankings are relative; a database's score can worsen even if its performance stays the same, simply because others improved.
- Benchmarks should be viewed critically, and results should align with real-world usage patterns.
- The article encourages running your own benchmarks tailored to specific workloads for more reliable insights.