OpenTelemetry for Go: Measuring overhead costs
a year ago
- #observability
- #opentelemetry
- #performance
- Observability comes at a cost, including instrumentation overhead like CPU, memory, and network usage.
- OpenTelemetry is the industry standard for collecting telemetry data (metrics, logs, traces) in a vendor-neutral way.
- A benchmark was conducted to measure OpenTelemetry's overhead in a Go HTTP server using Valkey (Redis fork).
- Test setup involved separate nodes for the app, Valkey, load generator, and observability (Coroot Community Edition).
- Without OpenTelemetry, the app handled 10,000 RPS with p95 latency ~5ms and CPU usage at 2 cores.
- With OpenTelemetry enabled, CPU usage increased by ~35%, memory by 5-8MB, and p99 latency rose to ~15ms.
- Network traffic increased by ~4MB/s due to telemetry data export.
- eBPF-based instrumentation offers a lightweight alternative with minimal overhead, suitable for high-throughput environments.
- Coroot's agent demonstrated low CPU usage (<0.3 cores) in metrics-only mode during the test.
- The choice between OpenTelemetry and eBPF depends on the need for detailed traces vs. low overhead.