Coding agents currently produce many false bug reports, with several agents scoring 10% or less true positive rates on Bismuth's benchmark.
2
Developers can improve results by giving agents scoped bug rules, managing context carefully, and using thinking models.
3
Even thinking models vary substantially between runs, so agents still do not provide a complete view of every bug in a codebase.
Summary
Ian Butler presents benchmark results on how well coding agents find and fix bugs. He says several popular agents have very low true positive rates and generate enough false positives to create alert fatigue. Cursor had a 97% false positive rate across more than 100 repositories and 1,200 issues in Bismuth's testing. Butler recommends adding scoped security and bug rules to agent configuration files, naming specific bug classes, requiring tests to pass after fixes, and managing context instead of letting important files disappear during compaction. He also recommends thinking models, which performed better at finding deeper bugs. Butler is honest about their limits: even when the total number of bugs found stays similar, the actual bugs identified can change from run to run. The talk ends with a brief description of Bismuth's automated code review and vulnerability scanning product.
Coding agents produce too many false bug reports to trust without checking
Butler says current agents have a low overall rate for finding real bugs and generate many false positives. Devon and Cursor had less than a 10% true positive rate for bug finding in the benchmark. Three of six agents scored at 10% or less across more than 900 reports. One agent reported 70 issues for a single task, all of which were false. Cursor had a 97% false positive rate across more than 100 repositories and more than 1,200 issues. Butler connects this to alert fatigue: developers become less willing to trust the agent, while real bugs can still reach production.
Scoped rules give agents specific bug classes to search for
Butler recommends using the rules file provided by each coding agent to add focused instructions about security issues and logical bugs. He suggests feeding the model security material such as the OWASP Top 10, which biases the model toward considering those problems while it reads the code. The rules should name specific bug classes rather than asking the agent to find bugs in general. His examples include auth bypasses, prototype pollution, and SQL injection. This makes the request more precise and primes the model to inspect the repository for those patterns.
Rules should require tests and validation before accepting a fix
A bug-finding instruction should also tell the agent to validate its proposed fix. Butler says the model should write tests and get them to pass before the change enters the codebase. In Bismuth's benchmark work across 100 repositories and thousands of issues, structured rules replaced vague requests to check for bugs. Butler says this produced higher-quality output and reduced the alert fatigue caused by broad, poorly defined scans. The practical point is to make the agent prove that a reported issue was fixed instead of accepting an untested edit.
Context loss makes agents miss bugs that cross files and components
Butler says agents struggle when they need to navigate broadly across a repository. After working for a while, they lose logical connections to code they have already read. When they reach context limits, they may summarize or compact files, and bug detection gets worse after that happens. He recommends feeding the agent diffs for changed code, keeping important files in the context window, and watching what gets removed during compaction. These steps help the model preserve cause-and-effect relationships across the codebase.
A component inventory helps an agent understand a codebase before searching for bugs
One technique Butler found effective was asking the agent to create a step-by-step inventory of the application. The inventory should index classes, variables, and how those elements are used across the codebase. Butler says that after producing this map, agents become more capable of finding bugs. This gives the model an explicit view of the relationships it needs to follow, instead of asking it to search a large repository with no structure. It directly addresses the navigation problems Butler observed in the benchmark.
Thinking models find deeper bugs than non-thinking models
Butler says thinking models performed significantly better at finding bugs. Their reasoning expands across several considerations in the codebase, then goes deeper into those considerations when looking for an issue. In practice, he says they found deeper bugs than non-thinking models across the benchmark. His advice is to reach for thinking models when using tools such as Claude Code or Cursor. The recommendation is based on the benchmark's observed difference in bug-finding performance, rather than on a claim that any model can inspect a repository completely.
Even better models change their findings from run to run
Thinking models still have a serious limitation. Across hundreds of repositories and thousands of issues, Butler says the top-line number of bugs found could remain the same while the identity of those bugs changed between runs. Agents therefore do not inspect a file as holistically as a human developer would. The result is high variability: repeating the scan can produce a different set of findings. Butler says users should not have to run an agent 100 times to obtain a complete bug breakdown, but he treats that as an unresolved problem.
Bismuth combines vulnerability scanning with automated code review workflows
Butler briefly describes Bismuth as an agentic coding solution that creates pull requests automatically. It connects with GitHub, GitLab, Jira, and Linear, scans for vulnerabilities, and provides reviews. He also mentions on-premises deployments. The talk points viewers to Bismuth's full benchmark, including its methodology, results, data set, and exploration tools. The product pitch follows the benchmark discussion and presents Bismuth as a way to apply automated bug and vulnerability review within existing development systems.
"Three out of six agents on our benchmark had a 10% or less true positive rate out of 900 plus reports."01:09
Who should watch
You use Cursor, Claude Code, or another coding agent and are seeing too many bug warnings to review them all.
Your agent works across a large repository and loses important context after summarizing or compacting files.
You want a practical way to write bug-focused rules and make AI-generated fixes pass tests before merging.
