ai code review

Bilingual

Engineers have stopped reviewing PRs
2 months ago
- Reviewing an AI coding session means evaluating the entire run, not just the diff, because key decisions happen during the session.
- Habit 1: Preserve the original brief verbatim in the pull request description, including corrections that changed scope.
- Habit 2: Keep the transcript retrievable using tools like tmux, script, or CLI exports (e.g., Claude Code's /export, Codex CLI session files).
- Habit 3: Review in five layers: brief, corrections, abandoned paths, overrides, and running behavior—each catches failures the diff hides.
- Habit 4: Make a second person the default reviewer for agent-assisted PRs to counter expectation bias.
- Session review takes 10-15 minutes and can be practiced with discipline; tools like AQ reduce friction by making the session a shared artifact.
Using OpenCode in CI/CD for AI pull request reviews
2 months ago
- OpenCode is an open-source CLI tool used in CI/CD pipelines to perform AI code reviews without granting third-party access to repositories.
- It addresses security concerns of existing AI code review SaaS tools by eliminating the need for OAuth or persistent third-party access to code repositories.
- The setup works with any YAML-based CI/CD system (e.g., GitHub Actions, GitLab CI, Bitbucket Pipelines) and involves cloning the repo, installing OpenCode, and running a prompt to analyze diffs.
- The AI review process uses a conservative prompt that focuses on critical issues (e.g., performance, security, validation) and avoids false positives by verifying concerns through code reading.
- Economics are favorable: OpenCode works with existing ChatGPT subscriptions (Plus/Pro/Business) at no extra cost, unlike per-seat or per-PR pricing from dedicated code review tools.
- OpenCode is provider-agnostic, allowing easy switching between AI models (e.g., from OpenAI to Minimax) without changing the pipeline configuration.
- Beyond PR reviews, OpenCode can be integrated into a Slack bot for on-demand code queries, enabling teams to ask questions about the codebase without opening an IDE.
- Users maintain control over their code as it only passes through their CI/CD runners, though code is sent to the AI provider for inference unless a local model is used for air-gapped setups.
- Future improvements include automated resolution of PR comments, specialized agents for tasks like UX auditing, and better handling of OpenAI key expiration via secrets management.
Formal verification might solve AI's review bottleneck
2 months ago
- A new software engineering paradigm combines formal verification and AI: humans specify requirements in a formal language like Lean, and AI writes code along with a machine-checkable proof of correctness, eliminating human review of generated code.
- AI alone faces a review bottleneck because it can write code at near-zero cost, but reviewing all that code becomes a new bottleneck; formal verification removes this by proving correctness automatically, with increased assurance as a side effect.
- The framework involves humans defining hard constraints (formal specification) and optimization objectives (benchmarks), while AI agents write code and proofs; human effort is minimal as constraints are proven and objectives measured automatically.
- Analogies to machine learning include risks like overfitting and specification gaming, where agents may exploit weaknesses in specs; tools like regularization and careful metric design are needed to manage these risks.
- A Hello World example (k-th smallest element) demonstrates the approach: Lean definitions capture correctness, agents fill in implementation and proof, and benchmarks optimize runtime; the agent improved from sorting to quickselect.
- A case study at powdr (autoprecompiles for zkVMs) shows success: 500 lines of Lean spec, benchmarks for optimization, and 100% AI-generated code and proofs without human review; circuit-size reduction matches previous implementation, and runtime improved significantly.
- The approach may generalize if writing specs is cheaper than maintaining code, and reusable spec libraries could help; integration via FFI allows incremental adoption, potentially shifting focus from code to requirements in software engineering.
Models are worse at reviewing their own code
2 months ago
- Models are worse at finding bugs in their own code; each model finds more bugs in code written by the other model.
- Models produce the same types of bugs they introduce most often, which they also miss during review.
- Claude models take a breadth-first approach (scope broadly), while GPT models go depth-first (investigate deeply).
- GPT often identifies bugs in reasoning traces but fails to post them due to conflicting instructions and post-training alignment.
- Opus produces many comments, including false positives, because it reviews with a human-like focus on intent and potential issues.
- Greptile built 'Model Inversion' feature: detect the coding agent author and route the review to a different model to catch more bugs.
- The tension between alignment (following instructions) and usefulness (reporting all bugs) is highlighted, especially with GPT.
Reviewing AI Code Is Not a Viable Argument
2 months ago
- The author expresses skepticism about LLM coding assistants, not due to IP, ecological, or quality issues, but based on scientific evidence questioning their ability to improve coding speed or quality.
- A key problem is the high error rate of LLM coding assistants, akin to an 'intern' producing incorrect code, with the proposed solution being thorough code review.
- Empirical research shows effective code reviews have limits: they should not exceed 1 hour or 400 lines of code (LOC) per hour to maintain efficiency.
- Applying these review limits to LLM-generated code means a developer can only review a few thousand LOC per day at best, imposing a productivity ceiling.
- Evidence suggests human reviewers of LLM-generated code are less effective—finding fewer defects while being overconfident—compared to reviewing human-written code.
- Proponents often promote LLM assistants for writing complex code like Bash scripts, which is hardest to review, exacerbating the review problem.
- To address skepticism, the author calls for empirical research on human reviewers' ability to detect defects in LLM-generated code and how reviewing such code differs from human code.
- The author is frustrated by proponents ignoring these issues and relying on anecdotal evidence instead of engaging with scientific data on code review practices.
Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark
2 months ago
- Benchmarks are often perceived as authoritative and objective but may lack depth if not scrutinized.
- The author critiques AI Code Review Bench methodology, arguing it lacks clear problem definition and splits AI code review into two problems: human assistance and machine verification.
- Human-focused code review prioritizes recommendations to optimize limited attention, while machine-focused code review emphasizes exhaustive analysis for repair agents.
- The paper acknowledges challenges like reviewer imperfections and Goodhart's Law but may overemphasize proxies like agreement with human reviewers over outcomes like reducing production failures.
- Organizational context and severity of issues are overlooked, with universal benchmarks potentially averaging different notions of software quality.
- Future code review systems should separate human comprehension (interface and recommendation design) from machine verification (exhaustive hardening and repair).
- Despite criticisms, the paper advances discussion with contributions like acknowledging human reviewer limitations and proposing metrics beyond recall and precision.
Gemini Code Assist will be shut down on July 17
3 months ago
- Gemini Code Assist on GitHub is an AI-powered tool that summarizes pull requests and provides in-depth code reviews to speed up the review process and improve code quality.
- Users can invoke Gemini Code Assist at any stage of a pull request and interact with it via comments, including using the /gemini tag to ask questions.
- There are two versions: an enterprise version (available via Google Cloud with higher quotas and centralized management) and a consumer version (which is being discontinued).
- Key differences include terms of service, daily quotas (enterprise offers 100+ pull requests/day vs. consumer's 33), and configuration management (enterprise allows cross-repository settings via Google Cloud).
- Gemini Code Assist excludes files in the .github/workflows directory from reviews to prevent insecure configurations and uses a Developer Connect connection in the us-east1 region for the enterprise version.

ai code review

Engineers have stopped reviewing PRs

Using OpenCode in CI/CD for AI pull request reviews

Formal verification might solve AI's review bottleneck

Models are worse at reviewing their own code

Reviewing AI Code Is Not a Viable Argument

Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark

Gemini Code Assist will be shut down on July 17