Lessons from Building an OTel Normalizer for GenAI
5 hours ago
- #AI Observability
- #OpenTelemetry
- #Data Normalization
- OpenTelemetry's GenAI semantic conventions are more like suggestions than a standard, leading to varied implementations.
- Groundcover's AI Observability solution supports two paths: SDK instrumentation and eBPF sensor, both aiming for unified output.
- Normalization involves handling three axes: Instrumentation SDKs, orchestration frameworks, and LLM providers, each with unique quirks.
- Four wire formats (A to D) are detected based on attribute prefixes, each requiring specific parsing logic.
- Key challenges include inconsistent attribute naming for models and tokens, and provider-specific token counting semantics.
- Cost normalization requires adjusting token counts for providers like Anthropic to account for cache usage.
- Provider names are spelled differently across SDKs, necessitating a mapping to canonical outputs.
- The normalizer absorbs complexity to allow DevOps teams to focus on root cause analysis without instrumentation overhead.