# Why Agent Hype Can Fall Short of Reality

Joel Becker, METR | AI Engineer CODE 2025 | 21:22

Source: https://www.youtube.com/watch?v=RhfqQKe22ZA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/why-agent-hype-can-fall-short-of-reality
Published: 2025-12-24
Tags: agents, benchmarks, coding-agents, evals, product-strategy

## TL;DR
- METR's human time horizon measure suggests that AI models can complete tasks that would take people much longer, with measured capability rising rapidly over calendar time.
- In METR's randomized trial, 16 highly experienced developers took 19% more time to complete real software tasks when AI tools were allowed.
- The gap may come from reliability demands, task messiness, developer context, task interdependence, and differences in how effectively the underlying models are elicited.

## Summary
Joel Becker compares two kinds of evidence about AI capability. METR's time horizon work measures how long a human would take to complete tasks that an AI can attempt, and shows strong progress across models. A field experiment gives a less flattering result. Sixteen experienced developers working on large open source repositories took 19% more time when they were allowed to use AI tools. Becker describes possible reasons for the difference. Developers already understood their repositories and often knew the solution, so directing an AI and checking its output could take longer than typing the change themselves. Real software work is messy, and code that passes unit tests may still be hard to review or merge. Becker also discusses reliability, task dependencies, small-study uncertainty, incentives, and differences in capability elicitation. He treats the field result as a serious puzzle rather than dismissing either source of evidence.

## Key ideas
### Benchmark scores are difficult to translate into useful capability claims
[01:50](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=110s)
Becker says benchmarks such as SWE-bench and GPQA have clear floors and human baselines, but the meaning of an intermediate score is uncertain. A model scoring 50% on GPQA is not straightforwardly half as capable as an expert, and matching a human baseline does not automatically mean matching expert performance in relevant work. Benchmarks also saturate quickly. They can go from providing some signal to providing little additional information as models improve, while creating harder tasks becomes difficult when capability is rising rapidly.

### Human time horizons compare AI performance with task difficulty
[03:06](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=186s)
METR first gathers baseline data from experienced people attempting tasks without prior context on those exact problems. The tasks range from small software problems in the SWAR suite to autonomous, tool-using tasks in HCAST and difficult machine learning engineering challenges in REBench. METR measures AI performance on the same tasks, then fits a curve against the time humans took. The time horizon is the human completion time where the model is predicted to succeed 50% of the time.

### Time horizon measurements show rapid progress across models
[05:01](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=301s)
Becker gives Claude 3 Opus a time horizon of roughly four minutes and o1-preview roughly 15 minutes in the example. Plotting these measurements against release dates produces a remarkably steady upward trend that looks close to exponential. He says later updates, including GPT-5.1 Codex Max, have continued to fit the straight-line prediction well. This evidence indicates that models can sometimes handle tasks far beyond Becker's own ability and that measured progress has been rapid.

### METR's developer trial tested AI in real repository work
[10:30](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=630s)
The randomized trial involved 16 experienced developers working on large, mature open source projects. Each developer completed about 16 real GitHub issues, randomly assigned to an AI-disallowed or AI-allowed condition. The allowed condition included Cursor and other AI tools, although developers were not required to use them. The repositories included the Haskell compiler, scikit-learn, and Hugging Face Transformers. Participants were typically among the top contributors, had contributed for about five years, and worked in codebases averaging more than a million lines.

### The field result was slower work despite strong expectations
[12:12](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=732s)
Outside observers expected developers to save about 40% of their time, while participants expected a gain of roughly 24% to 25%. After the study, participants estimated that AI had sped them up by about 20%. The measured result went in the opposite direction: developers took 19% more time when AI was allowed. Becker initially suspected an experimental mistake, then reviewed many hours of screen recordings and examined possible explanations. He also cautions that the study covered an unusual group of developers and was concentrated in March 2025.

### Real repository work demands context and reliability
[14:21](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=861s)
Becker says the developers already knew their repositories and often knew how to solve the issues, so asking an AI to make changes could cost more time than typing directly. Models may also work less well in large, complex repositories than on contained benchmark environments. Even when an AI produces useful code, developers need to check it, and they may need to repair it. Becker suggests that saving time may require answers that are correct around 95% to 99% of the time, rather than answers that are merely successful on a task some of the time.

### Several differences could reconcile benchmark and field evidence
[18:18](https://www.youtube.com/watch?v=RhfqQKe22ZA&t=1098s)
Becker lists several possible explanations without presenting one as settled. Benchmark scoring may reward passing tests while missing mergeability, maintainability, or other quality concerns. The human baselines in METR's task studies may have less context and skill than the developers in the trial. The tasks may differ in how messy they are, and Cursor's capability elicitation may have been less optimized than METR's agents. Tasks can also depend on one another: a human may need to understand how task A was completed before reliably doing task B, which limits the benefit of delegating only task A.

## Notable quotes
- "The punch line is that we find that developers are slowed down by 19%. They take 19% more time when AI is allowed relative to when AI is not allowed." (13:09)
- "You need to be getting the answers these problems that developers are putting in correct, something like 95, 99% of the time in order for developers to tab through and not spend lots of time verifying the AI's work." (18:22)
- "It's harder and harder to create benchmarks that have plenty of signal that might be informative to us about how capable models are for an extended period of time." (02:46)
- "These are extremely experienced developers working on extremely complex long-lived open source repositories." (16:00)

## Tools & references mentioned
- METR
- Model Evaluation and Threat Research
- SWE-bench
- GPQA
- HCAST
- SWAR suite
- REBench
- Claude 3 Opus
- Claude 3.7 Sonnet
- o1-preview
- GPT-5.1 Codex Max
- Cursor
- Haskell compiler
- scikit-learn
- Hugging Face Transformers

## Who should watch
- You are comparing benchmark gains with actual productivity and want to understand why the two can point in different directions.
- You build coding agents and need to think about verification costs, repository context, and dependencies between software tasks.
- You evaluate AI systems for economic impact and want evidence from a randomized developer study rather than model scores alone.

## Related talks

- [How METR Measures Long Tasks and Experienced Open Source Developer Productivity](https://aietalks.com/talks/how-metr-measures-long-tasks-and-experienced-open-source-developer-productivity) (Joel Becker, METR, 1:15:52)
- [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)
- [The Art & Science of Benchmarking Agents](https://aietalks.com/talks/the-art-science-of-benchmarking-agents) (Vincent Chen, Snorkel AI, 23:25)
- [Agentic Evaluations at Scale, For Everybody](https://aietalks.com/talks/agentic-evaluations-at-scale-for-everybody) (Nicholas Kang & Michael Aaron, Google DeepMind, 20:03)
- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)