# How METR Measures Long Tasks and Experienced Open Source Developer Productivity

Joel Becker, METR | AI Engineer CODE 2025 | 1:15:52

Source: https://www.youtube.com/watch?v=k1t2xyWMUdY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-metr-measures-long-tasks-and-experienced-open-source-developer-productivity
Published: 2026-01-19
Tags: agents, benchmarks, coding-agents, evals

## TL;DR
- METR's time-horizon measurements show rapid capability growth, but the evaluation tasks may eventually become too short to measure further progress.
- A randomized study of experienced open source developers found no clear speedup from Cursor, despite rising benchmark scores and developers' perceptions that AI made them faster.
- Real-world work exposes limits that benchmarks miss, including repository context, reliability, ambiguous goals, messy company data, and the need to maintain high-quality changes.

## Summary
Joel Becker explains how METR measures the length of tasks that AI systems can complete and why that measure matters for forecasts about automated AI research. He connects time-horizon growth to compute spending, while discussing limits from power, dollars, evaluation time, and hardware. The talk then examines METR's study of experienced open source developers, where AI access did not produce a clear productivity gain. Self-reported time estimates were unreliable, and the small sample made subgroup patterns such as a possible learning curve hard to trust. Becker and the audience discuss why mature open source repositories may be unusually demanding, how enterprise data creates problems for AI systems, and why real-world agents often fail at fuzzy goals. Becker argues for evidence beyond benchmarks, including natural-use transcripts and monitored versus unmonitored tasks. He ends by considering whether AI could automate chip production and where it might first improve fabrication, especially computational design work and yield detection.

## Key ideas
### Compute growth could eventually slow the rise of measured AI capabilities
[00:13](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=13s)
Becker begins with a simple model: if compute spending and AI time horizons are causally proportional, a slowdown in compute growth would also slow capability growth. He points to physical limits such as power, while saying dollars may be the more likely constraint because large companies and nation states can spend only so much. Under a standard economic assumption, he expects the relationship to hold while a software-only singularity is not possible. He also leaves room for an unpredictable technical advance, such as another transformer-style breakthrough, to change the curve.

### Time horizons will eventually run into the length of the evaluation tasks
[04:19](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=259s)
As time horizons keep doubling, evaluation tasks may become too short to distinguish stronger systems. Becker says METR is working on ways around this, although the approaches still feel early. He also separates model working time from human calendar time. Models often do successful work early in a task, while much of the elapsed time comes from the human-model iteration loop. That means reducing model completion time may matter less than improving reliability. At some point, the useful goal may be the same result in less time rather than a longer task-completion horizon.

### Developer productivity may have a learning curve, but the study does not explain the full gap
[06:49](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=409s)
A Meta presentation found a J-curve after developers received agents: productivity initially fell, then improved over a period Becker recalls as roughly three or six months. Becker agrees that tool experimentation can make people slower before later benefits appear. He doubts this explains the large difference seen in the METR results. Most study participants already used language models in their normal workflow, although about three quarters were unfamiliar with Cursor at the start. Watching the recordings, Becker did not see an advanced workflow that the developers were failing to use.

### People's estimates of task time are poor measures of actual speedup
[08:53](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=533s)
The discussion separates perceived productivity from measured time. The audience notes that Meta found people can accurately report how much more productive they feel, but they are usually wrong when asked how long a task took. Becker says this matters because proposed measures of AI-driven R&D acceleration often ask developers or researchers how much faster they became. He therefore puts little faith in those estimates. Perception still matters because developers may believe they are faster when they are not, which can shape both adoption and hype.

### The METR study found no reliable familiarity cutoff for experienced developers using Cursor
[14:10](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=850s)
METR's study involved 16 experienced open source developers working on mature projects. Becker shows plots split by prior AI and Cursor experience, but warns that the sample is too small for strong subgroup conclusions. Developers with more experience did not show a clear improvement over time. One apparent J-shaped pattern depended on conservatively coded usage hours. A developer who had logged 140 hours of Cursor use estimated only 50, placing them in a lower bucket. Moving that person could change the visual pattern, which is why Becker labels the evidence unclear. He also says the projects and developers are a narrow population that is difficult to generalize.

### Experienced open source work contains maintenance demands that benchmark tasks often omit
[21:08](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=1268s)
The study's repositories are mature open source projects with a high quality bar. Open source projects that are hard for new contributors to understand often do not survive, while enterprise code can persist because it supports a profitable business. The audience adds that maintainers may reject a pull request because it makes future maintenance harder, even if it gets the immediate feature working. Becker describes one striking case involving the Haskell compiler, where a maintainer may argue in review for hours until a change meets exact expectations. In the study, the median pull request required zero minutes of coding after review, suggesting that contributors often produced nearly acceptable changes on the first attempt.

### Messy enterprise data may limit AI's value for data science
[25:25](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=1525s)
The audience argues that data science may be less amenable to AI help than software engineering because company data is difficult to interpret. At LinkedIn, there are thousands of tables with 'impressions' in their names, so an analyst must discover which one is correct. The problem is not mainly writing SQL. Tables can contain changing meanings, inconsistent time fields, and undocumented cutoffs. A deployment analysis might require joining telemetry, team hierarchies, pull requests, and GitHub data that is absent from the deployment system. Becker agrees that tacit knowledge embedded inside companies may be important, while the audience says better data specifications and fixes at the source are often missing.

### Real-world agents need to handle fuzzy goals and interfaces suited to their abilities
[52:52](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=3172s)
Becker proposes studying transcripts from agents used in products such as Cursor because these records show what happens with real work rather than neatly packaged benchmark tasks. He also discusses AI Village, where agents use computers and try to accomplish loosely specified goals such as organizing an event or running a store. The agents often fail, although computer-use interfaces, model selection, and elicitation may explain part of the result. The audience suggests that models may be mismatched with interfaces designed for humans. Becker agrees that text or CLI-based tasks may be easier for current models, and says automating R&D may require models to scope work for themselves.

### Chip production could become a hardware bottleneck even if AI research is automated
[1:06:31](https://www.youtube.com/watch?v=k1t2xyWMUdY&t=3991s)
Becker questions whether AI research can be automated without automating chip design and production. Fixed hardware would limit the number of useful experiments, although a robot-and-AI economy could eventually remove that constraint. The discussion is cautious about robotics because manufacturing requires substantial human expertise, slow iteration, and expensive facilities. The audience notes that building a fabrication plant is very different from producing one-off robots. Becker mentions a large gap between language-model and robotics-model compute levels, and considers whether closing it could produce more capable robots. In chip fabrication, he expects near-term AI gains in computational mask design and detecting process parameters that threaten wafer yield.

## Notable quotes
- "Most of the time is spent in the human machine iteration loop." (05:47)
- "The one question you can't ask people on a survey is how long did a task take." (08:53)
- "I just do not see these advanced workflows that they're not accessing." (11:48)
- "The basic issue is it's a small sample size and there's also a lot of bias in the data set effectively." (18:19)
- "It is okay at doing very trivial things and it fails at all complex tasks." (30:00)

## Tools & references mentioned
- METR
- Meta
- Epoch
- Cursor
- SWE-bench
- AI Village
- Haskell compiler
- GitHub
- LinkedIn
- GPT-4
- GPT-5
- ChatGPT
- Harvey
- Nvidia
- Hazel Hopper
- Arjun Ramani

## Who should watch
- You are interpreting rising benchmark scores and want to know why they may not predict productivity for experienced developers.
- You are designing an AI evaluation and need measures that include reliability, maintenance work, ambiguous goals, or real company data.
- You are thinking about automated AI research and want to consider compute limits, robotics, chip production, and the gap between text tasks and physical work.

## Related talks

- [Why Agent Hype Can Fall Short of Reality](https://aietalks.com/talks/why-agent-hype-can-fall-short-of-reality) (Joel Becker, METR, 21:22)
- [Coding Evals: From Code Snippets to Codebases](https://aietalks.com/talks/coding-evals-from-code-snippets-to-codebases) (Naman Jain, Cursor, 18:08)
- [The Art & Science of Benchmarking Agents](https://aietalks.com/talks/the-art-science-of-benchmarking-agents) (Vincent Chen, Snorkel AI, 23:25)
- [Agentic Evaluations at Scale, For Everybody](https://aietalks.com/talks/agentic-evaluations-at-scale-for-everybody) (Nicholas Kang & Michael Aaron, Google DeepMind, 20:03)
- [Benchmarks: The Good, the Bad, and the Ugly](https://aietalks.com/talks/benchmarks-the-good-the-bad-and-the-ugly) (Ali Khial, G2i, 12:49)