How we verify Cleric’s production fixes
5 hours ago
- Production verification is necessary but difficult due to delayed feedback and unreliable self-assessment by the investigation agent.
- A separate 'auditor' agent critiques the investigation process, improving acknowledgment of uncertainty.
- Verification uses a separate agent per issue that reads findings but forms its own judgment from production evidence.
- Dynamic code generation creates checks that run periodically, with the agent re-invoked only when status changes or monitoring ends.
- The verifier focuses on deeper signals (defects) rather than just alert recovery, and uses a rubric refined through failure cases.
- Learning from experience includes avoiding compounding errors from the verifier's own memory and tightening the standard of proof.
- Outcome data shows most fixes are applied on threads without human reply, and recurring alert clusters often stop after a fix is applied.
- Counterfactual analysis and user actions provide labeled data to improve suggestions and confidence scoring.
- The approach generalizes beyond fixes to verify any code or configuration change in production.