2 months ago
- Describes a high-performance log processing pipeline designed for embedded devices, handling logs from over 100,000 devices.
- Introduces techniques like using /dev/kmsg for unified kernel and userspace logging, PRINTK_PERSIST for retaining logs across reboots, and a simple log receiver for reliability.
- Emphasizes processing unstructured logs into structured events flexibly, without requiring changes to log-producing software.
- Highlights cost-effectiveness, with logs stored cheaply in S3-like systems and processed by a single VM handling terabytes daily.
- Notes the system evolved into primary monitoring, detecting issues like network problems and hardware defects, and influenced a company startup.