LLM Observability in the Wild – Why OpenTelemetry Should Be the Standard
4 hours ago
- #OpenTelemetry
- #AI Agents
- #LLM Observability
- Chatwoot's AI agent 'Captain' faced production issues like responding in Spanish unexpectedly and incorrect responses.
- LLM observability is crucial for understanding AI decisions, retrieved documents, tool calls, and input/output at each step.
- OpenAI's native tracing is detailed but tightly coupled to its framework and lacks span filtering.
- New Relic supports OpenTelemetry but has a cumbersome UI for debugging.
- Phoenix follows OpenInference, offering rich AI-specific span types but lacks Ruby SDK and full OpenTelemetry compatibility.
- OpenTelemetry is industry-standard but lacks AI-specific span types, while OpenInference is AI-focused but less adopted and compatible.
- Ruby's lack of OpenInference SDK support complicates observability for Chatwoot.
- SigNoz advocates for OpenTelemetry-native LLM observability to avoid fragmentation and maintain coherence across the stack.
- Recommendations include picking one telemetry backbone, using LLM-specific libraries close to OpenTelemetry, and following the OTel GenAI working group.
- SigNoz is investing in OpenTelemetry-native LLM observability, providing clear dashboards and guidance for popular frameworks.