SM-100 tests software agents on maintenance work such as bug fixes, dependency upgrades, and migrations, which existing coding benchmarks largely miss.
2
Agents often find only a narrow subset of bugs and produce many false positives, while their patches are usually easier because the bugs they identify tend to be simple.
3
The strongest systems found only a minority of the benchmark's bugs, showing that agents still struggle with holistic code understanding, cross-file reasoning, and reliable bug triage.
Summary
Ian Butler and Nick Gregory introduce SM-100, a benchmark built from 100 validated bugs across 84 public repositories. It tests whether agents can find bugs without being told where they are, identify bugs when they are introduced in a pull request, avoid false positives, and produce fixes that do not break the codebase. The benchmark covers Python, TypeScript, JavaScript, and Go, with objective failures such as data loss, crashes, and security issues. Butler and Gregory report that a basic agent loop found some bugs but produced a 97% false positive rate. Bismuth found 10 of the benchmark's hidden bugs, while the next system found seven. Other agents generated hundreds or thousands of reports with low true-positive rates. The speakers argue that agents are much better at creating software than maintaining deployed software. They need broader and deeper reasoning, better navigation, and stronger program comprehension before engineers can rely on them for maintenance.
Existing coding benchmarks measure feature development, but software work also includes planning, code review, deployment, and maintenance. Butler and Gregory focus on maintenance tasks such as bug fixes, dependency upgrades, and migrations. These tasks still involve writing code, yet they require an agent to understand an existing system and reason about how parts of it connect. Finding a bug can require deeper understanding than implementing the original feature. The speakers say this makes maintenance a distinct task rather than a small variation on code generation.
Agents miss bugs because their reasoning is narrow
The speakers found that agents struggle to evaluate files and systems as a whole. A run may find one subset of bugs while missing others, and thinking models only partly improve this behavior. Agents can confirm bugs that a human developer would quickly reject, while missing bugs that seem obvious to a person. Patching is often easier when the agent has already identified a bug, but the bugs being found are usually simple. A successful patch therefore does not show that the agent can reason through complex maintenance work.
SM-100 contains 100 bugs gathered, triaged, validated, and classified across more than 84 public repositories. The benchmark includes explicit security or logical issues that can cause data loss or system crashes. It leaves out feature requests, optimization, formatting, and design choices because people can reasonably disagree about them. Each bug has metadata covering severity, the code context, the domain knowledge needed to find it, the difficulty of locating it, and its consequences. The benchmark spans Python, TypeScript, JavaScript, and Go.
The benchmark measures discovery, precision, introduction-time detection, and repair
For each system, SM-100 measures whether it can discover a hidden bug without prior knowledge, the false-positive rate of its reports, whether it can identify a bug in the pull request or commit that introduced it, and whether its proposed fix works without breaking the rest of the codebase. To make hidden-bug discovery practical, the evaluation gives the agent the files in a relevant subsystem instead of the entire repository. The subsystem is selected from files changed by the commit that introduced the bug, without describing the bug itself.
A basic implementation can give an agent shell access, search and replace, a bug-report tool, and a finish command, then run it in a loop. Butler and Gregory found that this setup discovered five or six bugs in their tests but produced a 97% false positive rate. They say useful performance depends on the model, the surrounding system, prompting, information presentation, and navigation strategy. Bismuth found 10 hidden bugs, compared with seven for the next solution, but the speakers still describe substantial room for improvement.
High report counts make agents unusable for review
Several agents produced between 900 and 1,300 reports while achieving true-positive rates between 3% and 10%. One agent produced 70 reports for a single issue, which the speakers say no engineer would realistically sift through. Some simple user-facing bugs were also missed. Butler gives a form example where a state flag was never reset, so the form appeared to remain filled after submission. Only Bismuth and Codex found that issue. The problem is therefore both recall and the accuracy of the information presented to engineers.
Agents inspect code through changing narrow slices
Across the systems tested, the speakers saw a common pattern: agents reason about a narrow set of possibilities and do not go deeply enough into those possibilities. The total number of bugs found per run stays roughly consistent, but the bugs found change from run to run. Butler and Gregory interpret this as evidence that agents are not inventorying a file holistically. Different context or model biases lead each run to inspect the same code in a different way. They argue that future systems need broader thinking chains, deeper reasoning, targeted search, cross-file reasoning, and better bug-pattern recognition.
Strong feature benchmarks do not predict maintenance performance
The speakers contrast agents' reported 60% to 80% scores on SWE-bench with their much weaker SM-100 results. In their view, agents can create software upfront but still struggle to manage and repair it after deployment. They say newer systems, including Bismuth and Codex, are beginning to produce tighter bands of more accurate findings and occasionally identify complex issues. Their conclusion is cautious: performance is improving, but the agents most frequently used today still carry a high risk of introducing bugs.
"Existing agents are able to create software upfront, but to manage and fix software after it's been deployed will be a major struggle as far as we see it."16:48
Who should watch
You are considering an autonomous coding agent for bug fixing, pull request review, or dependency work and need to understand how much noise it may produce.
Your team evaluates coding models on feature implementation but has no test for maintenance, regression risk, or false-positive bug reports.
You build agent infrastructure and want concrete failure modes around repository navigation, code comprehension, and hidden-bug discovery.
