observability

Bilingual

Instrumenting my espresso machine with OpenTelemetry
2 months ago
- The author treats an espresso machine as a distributed system, instrumenting it with OpenTelemetry to debug shot quality issues.
- Challenges include fitting OTLP protobuf on an ESP32, avoiding network I/O blocking the brew control loop, and ensuring clock accuracy via NTP before exporting.
- Derived metrics like puck resistance coefficient of variation are computed on-device to detect channeling without storing full timeseries.
- High-resolution shot curves are achieved using a ring buffer and concatenated protobuf messages to stay within memory limits.
- ClickHouse handles high-cardinality attributes (e.g., unique shot IDs) without TSDB-style explosion, enabling trace-to-metric joins.
- A Home Assistant MQTT receiver is added via a custom OpenTelemetry Collector to ingest house sensor data into the same backend.
- An LLM agent queries ClickHouse with espresso domain knowledge to analyze shots and provide actionable fixes, closing the feedback loop.
Clickhouse is winning the Observability Wars
2 months ago
- Logs are consistently the worst part of observability work due to volume, schema drift, and conflicting expectations between developers and non-technical consumers.
- ClickHouse's columnar storage allows queries to read only relevant columns, significantly reducing I/O and enabling 10–14x compression on observability data.
- At 1 TB/day, all major observability stacks (Elasticsearch, LGTM, Datadog, ClickHouse) are comparable in complexity and cost.
- At 5–10 TB/day, Elasticsearch requires Kafka, shard management, and multiple clusters; LGTM demands 180+ pods and dedicated teams; Datadog becomes financially untenable with six- to seven-figure monthly bills.
- ClickHouse scales linearly by simply adding shards, maintaining the same architecture, query language, and operational model regardless of data volume.
- The trade-off for ClickHouse is upfront schema design and query complexity, but this investment pays off as data grows by orders of magnitude without architectural shifts.
Let's break autovacuum in Postgres: reproducing failures to make it observable
2 months ago
- Autovacuum prevents bloat and transaction ID wraparound, but failures are silent and hard to monitor.
- Using raw dead tuple count is misleading; compute 'autovacuum pressure' (dead tuples / threshold) for a normalized signal.
- Coroot detects problems by combining pressure >2x and at least 512MiB of dead data, then identifies root cause via system views.
- Five common failure modes: autovacuum disabled on table, long transaction holding horizon, replication slot pinning horizon, all workers busy, vacuum throttled.
- Each failure has distinct detection fingerprints: reloptions, xmin age by holder, worker count, wait event.
- The tool provides specific findings with table name, cause, and fix, reducing guessing.
- The approach uses existing Postgres system views (pg_stat_user_tables, pg_stat_activity, pg_stat_progress_vacuum, etc.) with smart aggregation.
Show HN: Observability for Coding Agents and LLM Applications
2 months ago
- Telemetry.dev is an observability platform for AI and LLM apps, capturing every model call, tool step, and retrieval as OpenTelemetry traces with attached tokens, cost, latency, and errors.
- Built on OpenTelemetry, it supports any framework and language without lock-in; first trace can be set up in five minutes.
- Free plan offers 10,000 spans per month with no credit card required, 7-day retention, one project, and two seats.
- Real-time monitoring surfaces failures as they happen, with error type, status, and full trace attached.
- Key metrics include volume, cost, p95 latency, and errors, split by model, provider, and environment.
- Input and output capture is per-environment, with built-in secret redaction and custom redaction patterns.
- Traces can be searched by name, filtered by status, environment, framework, or session, and opened in full detail.
- Supports SDKs for TypeScript and Python, plus integrations with Vercel AI SDK, LangChain, and any OTLP/HTTP exporter.
- Cost is calculated server-side using real token usage against current pricing for 4,700+ models, with optional per-span override.
Have you heard? ClickHouse is winning the observability wars
3 months ago
- ClickHouse excels in observability at large scales (e.g., 10TB/day) while maintaining consistency and performance, unlike other tools like Elastic, LGTM, and Datadog that struggle.
- Observability built on columnar storage represents a fundamental improvement over the traditional three pillars model, offering better handling of high cardinality data and scalability.
- Many newer observability vendors use columnar storage but market themselves as cheaper alternatives to Datadog without highlighting the architectural advantages, obscuring innovation.
- The author argues that the value in observability comes from unified data (like traces or structured logs) rather than segmented pillars, which is threatened by Datadog's business model.
- ClickHouse's willingness to align with 'Observability 2.0' contrasts with other vendors, as they focus on solving customer problems rather than just competing on price.

observability

Instrumenting my espresso machine with OpenTelemetry

Clickhouse is winning the Observability Wars

Let's break autovacuum in Postgres: reproducing failures to make it observable

Show HN: Observability for Coding Agents and LLM Applications

Have you heard? ClickHouse is winning the observability wars