# How to Build an AI Code Review Agent

Daksh Gupta, Greptile | AI Engineer NYC 2026 | 18:27

Source: https://www.youtube.com/watch?v=4UtrRVtUN_g
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-build-an-ai-code-review-agent
Published: 2026-10-26
Tags: agents, coding-agents, computer-use, cost, evals

## TL;DR
- A webhook, sandbox, coding agent, and review prompt can produce a basic pull request reviewer, but it will miss bugs, misunderstand the codebase, cost heavily, and generate distracting nitpicks.
- Historical incident pull requests, targeted prompts, model inversion, and detailed codebase context give the reviewer better evidence about which bugs to find.
- Autonomous testing and routing smaller models to routine subtasks can expand coverage while reducing the cost of reviewing each pull request.

## Summary
Daksh Gupta explains how companies can build an internal AI code review agent, starting with a sandbox, a coding-agent harness, and a GitHub or GitLab webhook. That simple system produces feedback, but it misses bugs, lacks codebase context, generates nitpicks, and can cost $3 to $25 per pull request. Gupta recommends prompts aimed at the bug types a company has actually seen, evaluation against historical incident pull requests, and model inversion so code generated by Claude is reviewed by GPT, or the reverse. He also describes a codebase knowledge base, access to tickets and documentation, and a living wiki that records application and production context. For behavior that static review cannot catch, the agent should run the application in a dependency-rich sandbox and test flows with computer vision. Finally, smaller models should handle file selection, context gathering, and exploratory work, while frontier models handle the most important analysis.

## Key ideas
### A basic webhook-and-sandbox reviewer is easy to assemble but misses the problems that matter
[03:05](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=185s)
Gupta's starting design has a sandbox, an agent such as Codex or Claude Code, a GitHub or GitLab pull request webhook, and a prompt that asks for review comments. Open Code and Pi are examples of model-independent harnesses. The system can send a pull request to the agent and return comments through GitHub's review APIs. In practice, this simple setup misses bugs, misunderstands libraries and distributed codebases, and does not know the author's intent. Gupta says coding-agent reviews commonly cost between $3 and $25 per pull request. They also produce technically correct but annoying nitpicks, which can cause another coding agent to add unnecessarily defensive code.

### Prompts should target a company's recurring bug types and explicitly reject nitpicks
[06:02](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=362s)
The first improvement is a more deliberate prompt. Gupta recommends cataloging the bug types found in a company's outages and incidents, then asking the reviewer to look for those specific failures. The prompt also needs to define nitpicks and tell the agent not to report them. Examples include demanding extra layers of defense or criticizing a slightly imperfect error-handling style. Small prompt changes can cause large changes in comment volume, so comment count alone cannot show whether the reviewer improved. A reviewer can report that it found a bug, but it cannot prove that no bug remains. Gupta says this is especially difficult when catastrophic bugs are rare.

### Historical incident pull requests provide a practical test set for reviewer changes
[07:38](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=458s)
Companies that record the pull requests behind outages and incidents can use them as an evaluation set. Each version of the review agent should run against those pull requests, and the team can measure how often it catches the known failures. This gives prompt changes a more reliable basis than simply reading the comments produced on current work. The set must keep changing because new coding agents create new classes of mistakes. Gupta says roughly 80 or 90% of bugs repeat earlier mistakes, while increasingly capable code agents will also fail in novel ways. Smaller companies may not have enough pull requests to cover the full range of bug types.

### Reviewing generated code with a different model catches failures shared by the generating model
[09:02](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=542s)
Gupta describes model inversion as choosing a reviewer that differs from the model that wrote the code. He says Claude models are about three times more likely to produce one-off errors than GPT models, while GPT models are more likely to produce authentication bypasses. A model reviewing its own family of generated code can repeat the same failure patterns, so Claude-written code should be reviewed with GPT, and vice versa. Model fingerprints can come from branch names, pull request text, commit messages, telemetry, or internal reports. Gupta says Greptile sees a 10% higher catch rate when models are flipped. Identifying the generating model becomes harder when developers create branches or write code manually.

### A codebase index should give the reviewer structured relationships instead of making it grep everything
[10:59](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=659s)
Gupta argues that grep alone is an imprecise and expensive way to provide codebase context. Coding-agent traces often repeat grep, file reads, and cache writes for large amounts of file content. His approach builds an internal knowledge base that catalogs parts of the codebase and their relationships. For authentication, it might record how the system works, map the flows, show a diagram, and list every involved file. That context helps with bugs such as authentication bypasses, where the reviewer must know whether a particular section runs inside an authenticated environment. More selective retrieval also reduces the amount of irrelevant code loaded into the agent's context.

### Tickets, documentation, and a living wiki supply intent that code alone cannot provide
[12:50](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=770s)
A pull request can be technically valid while failing to do what its author intended. Gupta points out that short Slack requests often leave substantial ambiguity, so an agent may misunderstand the requested change. He recommends giving the agent access to Jira or Linear, plus Confluence, Notion, or another documentation system. These sources contain the tribal knowledge engineers use without consciously consulting it. Greptile combines them into a living wiki, which records codebase behavior, application information from logs and production telemetry, and roadmap information from documents and tickets. The wiki supplies context to the agent and makes code searches more precise by helping it avoid files that do not matter.

### End-to-end testing in a configured sandbox can find failures that source review cannot see
[14:26](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=866s)
Static inspection can miss behavior that appears only when the application runs. Gupta gives the example of a React change that looks correct in changed files but is hidden on an iPhone after rendering. The proposed system uses a configurable, dependency-rich virtual machine, a pull request branch, and a build script that can start the application. The agent then uses computer vision to exercise documented application flows. Teams need a flow map that describes paths such as authentication and sign-up. A vision model can detect that a button is technically present but hidden on an iPhone, or that it falls below the first screen. Gupta calls this technically difficult and says the build and environment setup are among the harder parts.

### Task-specific model routing lowers cost while preserving stronger models for important judgments
[16:05](https://www.youtube.com/watch?v=4UtrRVtUN_g&t=965s)
After adding context and autonomous testing, review costs can still reach $3 to $20 or more per pull request. Gupta recommends using frontier models for the most important tasks and smaller models for exploratory work, file reads, context gathering, and other routine subtasks. Greptile's illustrative split uses a small model to find context, a lightweight computer-use model to test flows, and a frontier model to analyze bugs and produce the review. In practice, Gupta says the system has about 20 subtasks with model choices tuned separately. He estimates that a simpler version can reach 70 to 80% of the way by replacing straightforward tasks with smaller models. Evaluation data is needed to detect when a cheaper model reduces bug-catching ability.

## Notable quotes
- "The first thing that's going to go wrong is you're going to miss a lot of bugs." (03:51)
- "You'll want to invert the model." (09:44)
- "We found that there's a 10% higher catch rate if you flip models." (10:45)
- "For the first time, I think end to end QA is nearly solved." (15:02)

## Tools & references mentioned
- Greptile
- Nvidia
- Coinbase
- Firecracker
- Codex
- Claude Code
- Open Code
- Pi
- GPT
- Claude
- GitHub
- GitLab
- Jira
- Linear
- Confluence
- Notion
- React
- Stripe

## Who should watch
- You are considering an internal pull request reviewer and need to understand the engineering work beyond connecting a webhook to a coding agent.
- Your current reviewer produces many comments but gives little confidence that it catches the failures that have caused incidents in your own codebase.
- You need to control review cost while adding codebase context, application testing, and different models for different review tasks.

## Editor's note

Daksh Gupta says each version of a code review agent should run against historical incident pull requests so the team can measure whether it catches known failures. Kitaru records an agent run with every model call and tool result, then replays the run with the same inputs and responses after a change. That gives teams a repeatable check without touching real systems.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [How to Kill the Code Review](https://aietalks.com/talks/how-to-kill-the-code-review) (Ankit Jain, Aviator, 16:26)
- [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45)
- [Fixing the PR Bottleneck](https://aietalks.com/talks/fixing-the-pr-bottleneck) (Matt Pocock, AI Hero, 22:35)
- [AI-Generated Code Is Already Competing With Human Code](https://aietalks.com/talks/ai-generated-code-is-already-competing-with-human-code) (Daksh Gupta, Greptile, 12:41)
- [AI Writes More PRs. Who Validates Them?](https://aietalks.com/talks/ai-writes-more-prs-who-validates-them) (Ali-Reza Adl-Tabatabai, Sonar, 11:34)
