Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark

7 hours ago

Benchmarks are often perceived as authoritative and objective but may lack depth if not scrutinized.
The author critiques AI Code Review Bench methodology, arguing it lacks clear problem definition and splits AI code review into two problems: human assistance and machine verification.
Human-focused code review prioritizes recommendations to optimize limited attention, while machine-focused code review emphasizes exhaustive analysis for repair agents.
The paper acknowledges challenges like reviewer imperfections and Goodhart's Law but may overemphasize proxies like agreement with human reviewers over outcomes like reducing production failures.
Organizational context and severity of issues are overlooked, with universal benchmarks potentially averaging different notions of software quality.
Future code review systems should separate human comprehension (interface and recommendation design) from machine verification (exhaustive hardening and repair).
Despite criticisms, the paper advances discussion with contributions like acknowledging human reviewer limitations and proposing metrics beyond recall and precision.

Hasty Briefsbeta