3 months ago
- AI root cause analysis (RCA) consists of two separate jobs: reasoning and the harness, which are often conflated.
- Reasoning involves connecting data points into a coherent story, while the harness involves data preparation and tool-calling.
- In a test scenario, with the harness removed, many models correctly identified a network delay caused by a Chaos Mesh experiment.
- Frontier models like Claude Opus 4.8 and GPT-5.5 passed cleanly, while some smaller models like Gemma 4 31B also succeeded.
- Failures were of two types: reasoning errors (missing the root cause) and formatting/instruction-following slips (fixable via harness).
- Cost per incident is low for reasoning-focused calls (~10k tokens), making top models affordable.
- The hard part of AI RCA is now the harness—preparing compact, relevant context—rather than the model's reasoning ability.