2 months ago
- Models are worse at finding bugs in their own code; each model finds more bugs in code written by the other model.
- Models produce the same types of bugs they introduce most often, which they also miss during review.
- Claude models take a breadth-first approach (scope broadly), while GPT models go depth-first (investigate deeply).
- GPT often identifies bugs in reasoning traces but fails to post them due to conflicting instructions and post-training alignment.
- Opus produces many comments, including false positives, because it reviews with a human-like focus on intent and potential issues.
- Greptile built 'Model Inversion' feature: detect the coding agent author and route the review to a different model to catch more bugs.
- The tension between alignment (following instructions) and usefulness (reporting all bugs) is highlighted, especially with GPT.