- MirrorCode is a benchmark co-developed with METR for long-horizon AI coding tasks, requiring complete end-to-end reimplementation of programs without access to the original source code.
- The benchmark features 25 target programs spanning Unix utilities, bioinformatics, cryptography, interpreters, compression, and other domains.
- It provides scale-aware evaluations with realistic inference budgets (up to $2,600 and 19 days per run), and is sandboxed and cheat-resistant with held-out tests.
- Frontier models can solve complex tasks; for example, Claude Opus 4.7 reimplemented the gotree bioinformatics toolkit in 14 hours at a cost of $251, a task estimated to take a human engineer 2–17 weeks.
- Data contamination is a caveat since the programs are open source, but performance on memorization screens suggests memorization is not the dominant factor.
- The leaderboard includes 30 tasks from the Medium and Large difficulty buckets, run with a budget of 10 billion tokens per attempt, and the benchmark is not fully solved.
- The scaffold and 22 out of 25 MirrorCode target programs are released as open source.