How we verify Cleric’s production fixes

5 hours ago

Production verification is necessary but difficult due to delayed feedback and unreliable self-assessment by the investigation agent.
A separate 'auditor' agent critiques the investigation process, improving acknowledgment of uncertainty.
Verification uses a separate agent per issue that reads findings but forms its own judgment from production evidence.
Dynamic code generation creates checks that run periodically, with the agent re-invoked only when status changes or monitoring ends.
The verifier focuses on deeper signals (defects) rather than just alert recovery, and uses a rubric refined through failure cases.
Learning from experience includes avoiding compounding errors from the verifier's own memory and tightening the standard of proof.
Outcome data shows most fixes are applied on threads without human reply, and recurring alert clusters often stop after a fix is applied.
Counterfactual analysis and user actions provide labeled data to improve suggestions and confidence scoring.
The approach generalizes beyond fixes to verify any code or configuration change in production.

Hasty Briefsbeta