Most "self-improving" AI agents don't improve
9 hours ago
- Most 'self-improving' AI agents in production lack true self-improvement; only 2 of 9 examined cases have verifiable self-improving loops.
- A self-improving loop's effectiveness depends on a verifier grounded in real-world data, which is hard, slow, and complex to build.
- Building verifiers is a key competitive opportunity; many AI projects avoid them due to difficulty.
- Key resources needed for verifier projects: (1) real-world data grounding, (2) expert feedback on failures and its cost.
- Replit's coding agent uses a verifier with simulated app tests and A/B testing, plus Telescope trace clustering.
- Tax AI agent improved from ~25% to 86% accuracy in six weeks via expert CPA feedback on each field.
- Auto-evals are a common but flawed shortcut; they fail due to criteria drift and reward hacking without real-world input.
- Before sponsoring a self-improving loop, align resources: real-world data source and expert feedback mechanism.