Using eBPF with OpenTelemetry
2 days ago
- #OpenTelemetry
- #Observability
- #eBPF
- eBPF enables full distributed tracing without adding instrumentation code by observing applications from the Linux kernel.
- Traditional OpenTelemetry instrumentation requires SDKs, exporters, and manual code wrapping, which is time-consuming for large or polyglot systems.
- eBPF-based tools like Beyla, Odigos, and Pixie automatically generate OpenTelemetry-compatible traces, metrics, and profiles.
- eBPF can observe network traffic, system calls, user functions, and language runtimes, converting this data into OpenTelemetry Protocol (OTLP) format.
- Deployment patterns include DaemonSet for Kubernetes, sidecar for per-pod isolation, and standalone for non-Kubernetes environments.
- Beyla is simple to set up and focuses on HTTP/gRPC auto-instrumentation, while Odigos provides comprehensive distributed tracing with context propagation.
- eBPF auto-instrumentation captures HTTP server/client requests, gRPC calls, database queries, DNS lookups, and TCP connections but lacks business context.
- Hybrid approaches combine eBPF for baseline coverage with manual instrumentation for business logic details, ensuring correlation via trace IDs.
- Performance overhead is low (1-5% CPU) but varies by workload; sampling and filtering can reduce impact.
- Limitations include Linux-only support, kernel version requirements, and lack of business context, necessitating manual instrumentation for custom attributes.