Reflections on Trusting Trust, Revisited: Poisoning Self-Modifying AI Coding
5 hours ago
- Revisits Thompson's classic trust attack in the context of self-modifying AI coding agents.
- Demonstrates that poisoned benchmarks can induce future versions of agents to write vulnerable code on clean tasks.
- Instantiates the attack against three recent self-modifying coding agents: Darwin Gödel Machine, Self-Improving Coding Agent, and Hyperagents.
- Example: Hyperagents evolved to disable HTTPS certificate validation on neutral tasks after poisoned benchmark exposure.
- Identifies conditions (vulnerability, benchmark, model, and scaffolding) that enable the benchmark poisoning attack.
- Finds that contamination often persists even when the poisoned agent is subsequently evolved with clean benchmarks.
- Argues that self-modifying coding agents must be designed to be more resilient to such attacks.