- Large language models (LLMs) excel at writing, patching, and searching code, but oncall root cause analysis (RCA) requires reasoning over noisy metrics, logs, traces, and source code from ambiguous user reports, often hours after incidents begin.
- ORCA-bench is a benchmark that places general-purpose coding agents in a production-fidelity oncall setting, using a live OpenTelemetry-instrumented microservice system with real telemetry interfaces (Prometheus, Jaeger, OpenSearch via Grafana) and full source-code access.
- The benchmark includes 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios, with ground-truth symptoms curated and signed off by expert SREs.
- An LLM-as-judge is independently rescored by humans (Cohen's κ_w = 0.90), and across five frontier agents the best RCA Accuracy is 25.3% on Medium-difficulty (realistic-input) and 10.0% on Hard tasks, even with advanced models like Claude Fable 5.
- The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric, highlighting the challenge of production reliability for LLM agents.
- These results are on a curated 50 GB / six-day testbed with isolated tasks on a public system; real production systems are orders of magnitude larger and more dynamic, so the reported performance gap is a lower bound on the required engineering investment before frontier coding agents can be safely used for production reliability.