- Performance improvements in benchmarks may not reflect live production due to noise in metrics.
- Multiple statistics (mean, median, p95, p99) can tell contradictory stories about the same data change.
- Cumulative distribution functions (CDFs) visualize the entire distribution, revealing shifts and crossing points.
- Rollout data over time shows emerging regression patterns that aggregates hide.
- Bimodality in latency often has a root cause, such as cache hits vs. misses based on response size.
- Jointplots can isolate populations by variables like response size to identify actionable causes.
- Multiple visualizations are needed to fully understand data instead of relying on a single number.