# Agents reported thousands of bugs, how many were real? Ian Butler & Nick Gregory, Bismuth | AI Engineer World's Fair 2025 | 18:39 Source: https://www.youtube.com/watch?v=wAQK7O3WGEE Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/agents-reported-thousands-of-bugs-how-many-were-real Published: 2025-06-03 Tags: agents, benchmarks, coding-agents, evals, testing ## TL;DR - SM-100 tests software agents on maintenance work such as bug fixes, dependency upgrades, and migrations, which existing coding benchmarks largely miss. - Agents often find only a narrow subset of bugs and produce many false positives, while their patches are usually easier because the bugs they identify tend to be simple. - The strongest systems found only a minority of the benchmark's bugs, showing that agents still struggle with holistic code understanding, cross-file reasoning, and reliable bug triage. ## Summary Ian Butler and Nick Gregory introduce SM-100, a benchmark built from 100 validated bugs across 84 public repositories. It tests whether agents can find bugs without being told where they are, identify bugs when they are introduced in a pull request, avoid false positives, and produce fixes that do not break the codebase. The benchmark covers Python, TypeScript, JavaScript, and Go, with objective failures such as data loss, crashes, and security issues. Butler and Gregory report that a basic agent loop found some bugs but produced a 97% false positive rate. Bismuth found 10 of the benchmark's hidden bugs, while the next system found seven. Other agents generated hundreds or thousands of reports with low true-positive rates. The speakers argue that agents are much better at creating software than maintaining deployed software. They need broader and deeper reasoning, better navigation, and stronger program comprehension before engineers can rely on them for maintenance. ## Key ideas ### Software maintenance needs its own evaluation [01:18](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=78s) Existing coding benchmarks measure feature development, but software work also includes planning, code review, deployment, and maintenance. Butler and Gregory focus on maintenance tasks such as bug fixes, dependency upgrades, and migrations. These tasks still involve writing code, yet they require an agent to understand an existing system and reason about how parts of it connect. Finding a bug can require deeper understanding than implementing the original feature. The speakers say this makes maintenance a distinct task rather than a small variation on code generation. ### Agents miss bugs because their reasoning is narrow [03:09](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=189s) The speakers found that agents struggle to evaluate files and systems as a whole. A run may find one subset of bugs while missing others, and thinking models only partly improve this behavior. Agents can confirm bugs that a human developer would quickly reject, while missing bugs that seem obvious to a person. Patching is often easier when the agent has already identified a bug, but the bugs being found are usually simple. A successful patch therefore does not show that the agent can reason through complex maintenance work. ### SM-100 excludes ambiguous issues [04:57](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=297s) SM-100 contains 100 bugs gathered, triaged, validated, and classified across more than 84 public repositories. The benchmark includes explicit security or logical issues that can cause data loss or system crashes. It leaves out feature requests, optimization, formatting, and design choices because people can reasonably disagree about them. Each bug has metadata covering severity, the code context, the domain knowledge needed to find it, the difficulty of locating it, and its consequences. The benchmark spans Python, TypeScript, JavaScript, and Go. ### The benchmark measures discovery, precision, introduction-time detection, and repair [07:35](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=455s) For each system, SM-100 measures whether it can discover a hidden bug without prior knowledge, the false-positive rate of its reports, whether it can identify a bug in the pull request or commit that introduced it, and whether its proposed fix works without breaking the rest of the codebase. To make hidden-bug discovery practical, the evaluation gives the agent the files in a relevant subsystem instead of the entire repository. The subsystem is selected from files changed by the commit that introduced the bug, without describing the bug itself. ### A simple agent loop produces too much noise [10:24](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=624s) A basic implementation can give an agent shell access, search and replace, a bug-report tool, and a finish command, then run it in a loop. Butler and Gregory found that this setup discovered five or six bugs in their tests but produced a 97% false positive rate. They say useful performance depends on the model, the surrounding system, prompting, information presentation, and navigation strategy. Bismuth found 10 hidden bugs, compared with seven for the next solution, but the speakers still describe substantial room for improvement. ### High report counts make agents unusable for review [11:32](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=692s) Several agents produced between 900 and 1,300 reports while achieving true-positive rates between 3% and 10%. One agent produced 70 reports for a single issue, which the speakers say no engineer would realistically sift through. Some simple user-facing bugs were also missed. Butler gives a form example where a state flag was never reset, so the form appeared to remain filled after submission. Only Bismuth and Codex found that issue. The problem is therefore both recall and the accuracy of the information presented to engineers. ### Agents inspect code through changing narrow slices [15:48](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=948s) Across the systems tested, the speakers saw a common pattern: agents reason about a narrow set of possibilities and do not go deeply enough into those possibilities. The total number of bugs found per run stays roughly consistent, but the bugs found change from run to run. Butler and Gregory interpret this as evidence that agents are not inventorying a file holistically. Different context or model biases lead each run to inspect the same code in a different way. They argue that future systems need broader thinking chains, deeper reasoning, targeted search, cross-file reasoning, and better bug-pattern recognition. ### Strong feature benchmarks do not predict maintenance performance [16:48](https://www.youtube.com/watch?v=wAQK7O3WGEE&t=1008s) The speakers contrast agents' reported 60% to 80% scores on SWE-bench with their much weaker SM-100 results. In their view, agents can create software upfront but still struggle to manage and repair it after deployment. They say newer systems, including Bismuth and Codex, are beginning to produce tighter bands of more accurate findings and occasionally identify complex issues. Their conclusion is cautious: performance is improving, but the agents most frequently used today still carry a high risk of introducing bugs. ## Notable quotes - "We found agents struggle with holistic evaluation of files and systems, only finding subsets of bugs per run." (03:09) - "Basic implementations are trivial. Give it a shell tool, search and replace, think, report bug, a finish tool, put it in a loop, and press play." (10:24) - "The highest popular agent score outside of us who are just launching now scored 7% on SM 100." (14:03) - "No engineer is going to look through 70 bugs and be like, "Oh yeah, that one for sure is the right one."" (14:28) - "Existing agents are able to create software upfront, but to manage and fix software after it's been deployed will be a major struggle as far as we see it." (16:48) ## Tools & references mentioned - Bismuth - SM-100 - Base10 - DeepSeek R1 - Llama 4 Maverick - HumanEval - Polyglot benchmark - LiveCodeBench - Claude Code - Codex - Devin - Cursor Agent - Cognition - SWE-bench - Anthropic - OpenAI - Google Vertex AI - Sonnet 4 - o3 ## Who should watch - You are considering an autonomous coding agent for bug fixing, pull request review, or dependency work and need to understand how much noise it may produce. - Your team evaluates coding models on feature implementation but has no test for maintenance, regression risk, or false-positive bug reports. - You build agent infrastructure and want concrete failure modes around repository navigation, code comprehension, and hidden-bug discovery. ## Related talks - [How to Improve Your Vibe Coding](https://aietalks.com/talks/how-to-improve-your-vibe-coding) (Ian Butler, Bismuth, 07:30) - [Your agent is blindfolded](https://aietalks.com/talks/your-agent-is-blindfolded) (Johan Lajili, Poolside AI, 09:58) - [Benchmarking Coding Agents on New vs Legacy Codebases](https://aietalks.com/talks/benchmarking-coding-agents-on-new-vs-legacy-codebases) (Denys Linkov, Wisedocs, 18:08) - [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00) - [Why Agent Hype Can Fall Short of Reality](https://aietalks.com/talks/why-agent-hype-can-fall-short-of-reality) (Joel Becker, METR, 21:22)