Bilingual

Gemini Code Assist will be shut down on July 17
18 days ago
- Gemini Code Assist on GitHub is an AI-powered tool that summarizes pull requests and provides in-depth code reviews to speed up the review process and improve code quality.
- Users can invoke Gemini Code Assist at any stage of a pull request and interact with it via comments, including using the /gemini tag to ask questions.
- There are two versions: an enterprise version (available via Google Cloud with higher quotas and centralized management) and a consumer version (which is being discontinued).
- Key differences include terms of service, daily quotas (enterprise offers 100+ pull requests/day vs. consumer's 33), and configuration management (enterprise allows cross-repository settings via Google Cloud).
- Gemini Code Assist excludes files in the .github/workflows directory from reviews to prevent insecure configurations and uses a Developer Connect connection in the us-east1 region for the enterprise version.
Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark
9 days ago
- Benchmarks are often perceived as authoritative and objective but may lack depth if not scrutinized.
- The author critiques AI Code Review Bench methodology, arguing it lacks clear problem definition and splits AI code review into two problems: human assistance and machine verification.
- Human-focused code review prioritizes recommendations to optimize limited attention, while machine-focused code review emphasizes exhaustive analysis for repair agents.
- The paper acknowledges challenges like reviewer imperfections and Goodhart's Law but may overemphasize proxies like agreement with human reviewers over outcomes like reducing production failures.
- Organizational context and severity of issues are overlooked, with universal benchmarks potentially averaging different notions of software quality.
- Future code review systems should separate human comprehension (interface and recommendation design) from machine verification (exhaustive hardening and repair).
- Despite criticisms, the paper advances discussion with contributions like acknowledging human reviewer limitations and proposing metrics beyond recall and precision.
Reviewing AI Code Is Not a Viable Argument
3 days ago
- The author expresses skepticism about LLM coding assistants, not due to IP, ecological, or quality issues, but based on scientific evidence questioning their ability to improve coding speed or quality.
- A key problem is the high error rate of LLM coding assistants, akin to an 'intern' producing incorrect code, with the proposed solution being thorough code review.
- Empirical research shows effective code reviews have limits: they should not exceed 1 hour or 400 lines of code (LOC) per hour to maintain efficiency.
- Applying these review limits to LLM-generated code means a developer can only review a few thousand LOC per day at best, imposing a productivity ceiling.
- Evidence suggests human reviewers of LLM-generated code are less effective—finding fewer defects while being overconfident—compared to reviewing human-written code.
- Proponents often promote LLM assistants for writing complex code like Bash scripts, which is hardest to review, exacerbating the review problem.
- To address skepticism, the author calls for empirical research on human reviewers' ability to detect defects in LLM-generated code and how reviewing such code differs from human code.
- The author is frustrated by proponents ignoring these issues and relying on anecdotal evidence instead of engaging with scientific data on code review practices.
Models are worse at reviewing their own code
6 hours ago
- Models are worse at finding bugs in their own code; each model finds more bugs in code written by the other model.
- Models produce the same types of bugs they introduce most often, which they also miss during review.
- Claude models take a breadth-first approach (scope broadly), while GPT models go depth-first (investigate deeply).
- GPT often identifies bugs in reasoning traces but fails to post them due to conflicting instructions and post-training alignment.
- Opus produces many comments, including false positives, because it reviews with a human-like focus on intent and potential issues.
- Greptile built 'Model Inversion' feature: detect the coding agent author and route the review to a different model to catch more bugs.
- The tension between alignment (following instructions) and usefulness (reporting all bugs) is highlighted, especially with GPT.

Hasty Briefsbeta

#ai code review

Gemini Code Assist will be shut down on July 17

Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark

Reviewing AI Code Is Not a Viable Argument

Models are worse at reviewing their own code