METR's human time horizon measure suggests that AI models can complete tasks that would take people much longer, with measured capability rising rapidly over calendar time.
2
In METR's randomized trial, 16 highly experienced developers took 19% more time to complete real software tasks when AI tools were allowed.
3
The gap may come from reliability demands, task messiness, developer context, task interdependence, and differences in how effectively the underlying models are elicited.
Summary
Joel Becker compares two kinds of evidence about AI capability. METR's time horizon work measures how long a human would take to complete tasks that an AI can attempt, and shows strong progress across models. A field experiment gives a less flattering result. Sixteen experienced developers working on large open source repositories took 19% more time when they were allowed to use AI tools. Becker describes possible reasons for the difference. Developers already understood their repositories and often knew the solution, so directing an AI and checking its output could take longer than typing the change themselves. Real software work is messy, and code that passes unit tests may still be hard to review or merge. Becker also discusses reliability, task dependencies, small-study uncertainty, incentives, and differences in capability elicitation. He treats the field result as a serious puzzle rather than dismissing either source of evidence.
Benchmark scores are difficult to translate into useful capability claims
Becker says benchmarks such as SWE-bench and GPQA have clear floors and human baselines, but the meaning of an intermediate score is uncertain. A model scoring 50% on GPQA is not straightforwardly half as capable as an expert, and matching a human baseline does not automatically mean matching expert performance in relevant work. Benchmarks also saturate quickly. They can go from providing some signal to providing little additional information as models improve, while creating harder tasks becomes difficult when capability is rising rapidly.
Human time horizons compare AI performance with task difficulty
METR first gathers baseline data from experienced people attempting tasks without prior context on those exact problems. The tasks range from small software problems in the SWAR suite to autonomous, tool-using tasks in HCAST and difficult machine learning engineering challenges in REBench. METR measures AI performance on the same tasks, then fits a curve against the time humans took. The time horizon is the human completion time where the model is predicted to succeed 50% of the time.
Time horizon measurements show rapid progress across models
Becker gives Claude 3 Opus a time horizon of roughly four minutes and o1-preview roughly 15 minutes in the example. Plotting these measurements against release dates produces a remarkably steady upward trend that looks close to exponential. He says later updates, including GPT-5.1 Codex Max, have continued to fit the straight-line prediction well. This evidence indicates that models can sometimes handle tasks far beyond Becker's own ability and that measured progress has been rapid.
METR's developer trial tested AI in real repository work
The randomized trial involved 16 experienced developers working on large, mature open source projects. Each developer completed about 16 real GitHub issues, randomly assigned to an AI-disallowed or AI-allowed condition. The allowed condition included Cursor and other AI tools, although developers were not required to use them. The repositories included the Haskell compiler, scikit-learn, and Hugging Face Transformers. Participants were typically among the top contributors, had contributed for about five years, and worked in codebases averaging more than a million lines.
The field result was slower work despite strong expectations
Outside observers expected developers to save about 40% of their time, while participants expected a gain of roughly 24% to 25%. After the study, participants estimated that AI had sped them up by about 20%. The measured result went in the opposite direction: developers took 19% more time when AI was allowed. Becker initially suspected an experimental mistake, then reviewed many hours of screen recordings and examined possible explanations. He also cautions that the study covered an unusual group of developers and was concentrated in March 2025.
Real repository work demands context and reliability
Becker says the developers already knew their repositories and often knew how to solve the issues, so asking an AI to make changes could cost more time than typing directly. Models may also work less well in large, complex repositories than on contained benchmark environments. Even when an AI produces useful code, developers need to check it, and they may need to repair it. Becker suggests that saving time may require answers that are correct around 95% to 99% of the time, rather than answers that are merely successful on a task some of the time.
Several differences could reconcile benchmark and field evidence
Becker lists several possible explanations without presenting one as settled. Benchmark scoring may reward passing tests while missing mergeability, maintainability, or other quality concerns. The human baselines in METR's task studies may have less context and skill than the developers in the trial. The tasks may differ in how messy they are, and Cursor's capability elicitation may have been less optimized than METR's agents. Tasks can also depend on one another: a human may need to understand how task A was completed before reliably doing task B, which limits the benefit of delegating only task A.
"You need to be getting the answers these problems that developers are putting in correct, something like 95, 99% of the time in order for developers to tab through and not spend lots of time verifying the AI's work."18:22
Who should watch
You are comparing benchmark gains with actual productivity and want to understand why the two can point in different directions.
You build coding agents and need to think about verification costs, repository context, and dependencies between software tasks.
You evaluate AI systems for economic impact and want evidence from a randomized developer study rather than model scores alone.
