# AI-powered entomology: Lessons from millions of AI code reviews Tomas Reimers, Graphite | AI Engineer World's Fair 2025 | 10:21 Source: https://www.youtube.com/watch?v=TswQeKftnaw Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/ai-powered-entomology-lessons-from-millions-of-ai-code-reviews Published: 2025-07-22 Tags: agents, coding-agents, evals, testing ## TL;DR - AI code review works best when it focuses on bugs and other comments that developers actually want to receive. - Graphite evaluates its reviewer with reactions and by measuring whether comments lead to code changes. - Graphite reports that fewer than 4% of Diamond comments receive downvotes, while 52% lead to a code change within the pull request. ## Summary Tomas Reimers describes what Graphite learned from building Diamond, its AI-powered code reviewer. The team found that AI can catch real failures, such as an uninstantiated database object that crashed a server or arithmetic that divided by a negative number. It also produces technically correct suggestions that developers find irritating, such as requests to add comments, extract functions, or add tests. Reimers presents code review comments on two axes: what an LLM can detect and what developers want to receive. Tribal knowledge remains difficult for AI, while code-cleanliness advice is often easy for AI to produce but unwelcome. Graphite uses comment upvotes and downvotes to detect hallucinations, then measures whether comments result in code changes. Human comments lead to changes about 50% of the time, while Diamond reached 52% by March after prompt changes. ## Key ideas ### AI can find bugs that cause real failures [00:58](https://www.youtube.com/watch?v=TswQeKftnaw&t=58s) Graphite began asking AI to review pull requests after seeing both AI-generated code and bugs increase. Reimers gives two examples. In Graphite's own codebase, a database object was returned without being instantiated, which would crash the server in certain cases. Another issue, found by the bot and shared on Twitter, involved math around border radiuses that could divide by a negative number and crash the front end. These examples convinced the team that AI could identify genuine bugs, even though its overall review output was initially inconsistent. ### AI code review has a usefulness problem beyond correctness [01:48](https://www.youtube.com/watch?v=TswQeKftnaw&t=108s) Early experiments also produced comments such as updating code to match another implementation, claims that CSS worked differently, or advice to revert code because an older version did things another way. The team then saw a second class of frustrating comments: asking developers to document a class, extract logic into a function, or add tests. Reimers says these comments are technically correct, but developers judge them based on whether the suggestion is welcome in that codebase. A comment can be valid and still waste the reviewer's attention. ### The useful comment space depends on both ability and acceptance [02:22](https://www.youtube.com/watch?v=TswQeKftnaw&t=142s) Graphite organized review comments around two questions. Can the LLM catch this kind of issue, and do humans want to receive this kind of comment from an LLM? Bugs, performance concerns, security concerns, accidental commits, documentation mismatches, and some stylistic changes fall into the useful area. Tribal knowledge is difficult because the explanation for an old design decision often exists only in senior developers' heads. Code cleanliness and generic best-practice advice are usually within the model's ability, but developers often do not want those comments. ### Graphite used a large comment sample to build its categories [04:00](https://www.youtube.com/watch?v=TswQeKftnaw&t=240s) The team took 10,000 comments from its own codebase and open-source codebases and gave them to several LLMs for categorization. They repeated the categorization and summarized the results into a chart of comment types. Reimers says the chart separates ordinary bugs from accidental commits, performance and security concerns, documentation, and style changes. It also shows comments that AI can produce but people do not want, alongside comments people value but AI cannot yet infer from the available context. ### More context may expand what AI can review well [05:33](https://www.youtube.com/watch?v=TswQeKftnaw&t=333s) After categorizing comments, Graphite changed its prompts so the LLM would focus on issues within its capabilities and comments developers wanted to receive. Users responded more positively. The team still needed to check whether this remained true as models changed, such as moving from Claude Sonnet to Claude 4 or Opus, and as they supplied more context. Reimers says the useful area might grow with more codebase history, style rules, and other context, but the system has to measure whether it is actually staying there. ### Reactions reveal when the reviewer exceeds its abilities [07:27](https://www.youtube.com/watch?v=TswQeKftnaw&t=447s) Graphite added upvotes and downvotes to Diamond's comments. Reimers treats downvote spikes as a signal that the model is hallucinating or trying to review work beyond its capabilities. The team can then reduce the scope of what it asks the LLM to do. This metric addresses whether the model can catch an issue, while the separate question of whether developers want the comment requires measuring what happens after the review. ### A review comment matters when it changes the code [08:32](https://www.youtube.com/watch?v=TswQeKftnaw&t=512s) Graphite measures how often comments lead to the change they describe within the same pull request. Reimers says only about 50% of human comments lead to changes. That does not mean the rest are useless: some are handled in a follow-up, some are advice for future work, and some are preferences where disagreement is acceptable. Diamond reached 52% by March. Reimers presents this as evidence that careful prompting can bring AI comments to at least the level of actionability seen in human reviews. ## Notable quotes - "Maybe AI is creating the bugs, but can it also find the bugs?" (00:58) - "The question is whether it's welcome in the codebase." (05:55) - "Only about 50% of human comments lead to changes." (08:47) - "As of March we're at 52%." (09:49) ## Tools & references mentioned - Graphite - Diamond - Claude - Claude 4 - Opus - Sonnet - GitHub ## Who should watch - You are evaluating an AI code reviewer and need metrics for usefulness beyond raw bug-finding accuracy. - Your team is getting too many automated comments about style, tests, or refactoring and wants a way to control their scope. - You want to measure whether review comments lead to changes rather than counting every generated suggestion. ## Related talks - [How to Kill the Code Review](https://aietalks.com/talks/how-to-kill-the-code-review) (Ankit Jain, Aviator, 16:26) - [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45) - [The State of AI Code Quality: Hype vs Reality](https://aietalks.com/talks/the-state-of-ai-code-quality-hype-vs-reality) (Itamar Friedman, Qodo, 21:15) - [Beyond the Prototype: Using AI to Write High-Quality Code](https://aietalks.com/talks/beyond-the-prototype-using-ai-to-write-high-quality-code) (Josh Albrecht, Imbue, 17:59) - [The Last Human Code Review: Building Trust in AI-Generated Code](https://aietalks.com/talks/the-last-human-code-review-building-trust-in-ai-generated-code) (Itamar Friedman, Qodo, 18:54)