- Gauntlet, an open-source multi-agent pipeline, evaluated deep technical comprehension of computer architecture papers, surpassing human analysis in most cases.
- In evaluations on 20 recent ISCA/HPCA papers, Gauntlet was preferred over human analysis in 15 out of 20 comparisons, with significant advantage in Critical Rigor.
- The multi-agent structure, particularly the synthesis pass, was key to Gauntlet's performance, outperforming a single-agent approach on 96% of papers.
- Human analyses were preferred only in aspects like trust and usefulness, often due to issues like confident but wrong claims or unprioritized breadth.
- All analyses, scores, and the evaluation rubric are released as a community resource for further research.