Logging Sucks
4 months ago
- #observability
- #logging
- #debugging
- Logging is fundamentally broken in modern distributed systems.
- Traditional logs lack context and are optimized for writing, not querying.
- String search in logs is inefficient due to inconsistent formats and missing relationships.
- Structured logging is necessary but not sufficient for effective debugging.
- Wide events (canonical log lines) provide comprehensive context per request.
- OpenTelemetry is a delivery mechanism but doesn't solve what to log or add business context.
- High-cardinality and high-dimensionality data make logs useful for debugging.
- Tail sampling helps manage costs by keeping critical events (errors, slow requests, VIP users).
- Wide events transform debugging from archaeology to analytics.
- Modern tools support high-cardinality data, making wide events practical.