# What Data from 20m Pull Requests Reveal About AI Transformation Nick Arcolano, Jellyfish | AI Engineer CODE 2025 | 17:57 Source: https://www.youtube.com/watch?v=WqZq8L-v9pA Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/what-data-from-20m-pull-requests-reveal-about-ai-transformation Published: 2025-11-24 Tags: agents, benchmarks, coding-agents, developer-experience, team-adoption ## TL;DR - Interactive AI coding tools are associated with roughly twice the PR throughput and 24% shorter cycle times when adoption rises from zero to full adoption. - Autonomous coding agents remain at an early stage, with fewer than half of companies trying them and less than 2% of merged PRs involving them. - Code architecture changes the outcome: centralized and balanced architectures show roughly 4x throughput gains, while highly distributed architectures show no meaningful relationship between adoption and throughput. ## Summary Nick Arcolano uses Jellyfish data from about 20 million pull requests, 200,000 developers, and 1,000 companies to describe how AI coding tools are being used. The data covers June 2024 onward and combines coding-tool activity with source control and task-management data. Developer adoption rose from a median of about 22% to nearly 90% among the tracked companies. Autonomous agents remain rare and are mostly being tested. Interactive tools correlate with a roughly 2x increase in PR throughput and a 24% reduction in cycle time from zero to full adoption. PRs also become 18% larger, while bug rates and revert rates show no statistically significant relationship with adoption so far. The gains vary sharply with code architecture. Centralized and balanced architectures show trends closer to 4x, while highly distributed architectures show no meaningful gain. Arcolano attributes this to the difficulty of giving tools context across repositories. ## Key ideas ### Jellyfish combines tool, repository, and planning data to study AI adoption [00:58](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=58s) Jellyfish combines usage from tools such as Copilot, Cursor, Claude Code, autonomous agents, and PR review bots with GitHub data and task-management systems such as Linear and Jira. This lets the company connect AI usage with the codebase and the goal of the work. The analysis covers about 20 million pull requests written and merged by roughly 200,000 developers across about 1,000 companies. The data was collected for more than a year, with the talk examining results from June 2024 onward. ### The share of code generated with AI has risen quickly, but developer usage is the better adoption measure [02:14](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=134s) Among a tracked cohort, only about 2% of companies generated at least half of their code with AI at the start of the period. By the latest month in the study, nearly half did. Arcolano prefers developer adoption because it captures behavior change more directly and correlates more closely with productivity outcomes. Jellyfish measures the fraction of time developers use AI tools while coding. Median company adoption rose from about 22% last summer to close to 90%. ### Autonomous coding agents are still mostly in trials [04:32](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=272s) Interactive tools account for most of the adoption data. Fully autonomous agents are much less common. Jellyfish saw only about 44% of companies do anything with autonomous agents during the previous three months. Most of that activity involved trying and experimenting rather than production use. Autonomous-agent work made up less than 2% of the millions of pull requests merged during that period. Arcolano describes this as an early stage rather than a mature deployment pattern. ### Full adoption of interactive tools correlates with twice the PR throughput [05:54](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=354s) Arcolano uses PR throughput as a practical productivity measure, while noting that its absolute level depends on work scope and architecture. The analysis compares company-week snapshots by AI adoption rate and average PRs merged per engineer. The average trend is about a 2x change from zero to full adoption of AI coding tools. Cycle time, measured from the first commit in a pull request until merge, also falls as adoption rises. The average trend is a 24% decrease from zero to full adoption. ### AI-assisted PRs become larger without an observed rise in bugs or reverts [08:57](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=537s) Teams with full adoption push pull requests that are about 18% larger in net lines of code added. The increase comes more from additions than deletions, and the average number of files touched stays about the same. Arcolano says this suggests more code within the same files rather than changes spread across more parts of the codebase. Jellyfish has not found a statistically significant relationship between AI adoption and bug-ticket creation or PR reverts. Teams are resolving more bugs because they use AI heavily on well-scoped bug tickets. ### Adoption has to reach scale before productivity gains appear [10:58](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=658s) Arcolano's first explanation for weak results is simple: teams need people using the tools at scale before gains appear. Some companies may still have low adoption because of technical, organizational, or cultural barriers. Others may have high adoption and still see weaker results. For those teams, he points to several possible areas for investigation and focuses on code architecture as one especially strong factor in the data. ### Code architecture changes the size of the AI productivity gain [11:57](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=717s) Jellyfish measures architecture with active repositories per engineer, meaning the number of distinct repositories where a typical engineer pushes code in a week. The measure is normalized by engineer count, so it can compare companies of different sizes. The study groups companies into centralized, balanced, distributed, and highly distributed regimes. Centralized and balanced architectures show trends closer to 4x PR-throughput gains with adoption. Distributed architectures follow the overall 2x trend. Highly distributed architectures show essentially no correlation between adoption and PR throughput. ### Repository boundaries create a context problem for current tools [14:38](https://www.youtube.com/watch?v=WqZq8L-v9pA&t=878s) Arcolano connects the weak results in highly distributed architectures to context. Most current tools work best inside one repository, while understanding a product may require context from several repositories. The relationships between repositories, systems, and products are often poorly documented and may remain in the heads of senior engineers. Coding tools and agents cannot easily use that information. Teams therefore need context engineering before highly distributed architectures can benefit fully. Arcolano says this result could change if cross-repository context improves and autonomous agents become reliable at scale. ## Notable quotes - "We only see about 44% of companies have done anything with autonomous agents at all in the past 3 months." (05:09) - "The average trend here is about a 2x change as you go from zero to full adoption." (07:14) - "The average trend is a 24% decrease in cycle times as you go from 0% to 100% adoption of AI coding tools." (08:11) - "What's really interesting is this highly distributed case. There's essentially no correlation here between AI adoption and PR throughput." (14:18) - "A big part of what you're seeing here is really the problem of context." (14:38) ## Tools & references mentioned - Jellyfish - GitHub - Copilot - Cursor - Claude Code - Devin - Codex - Linear - Jira ## Who should watch - Engineering leaders who are paying for AI coding tools and need measures beyond headline usage or lines of code. - Teams that have adopted AI tools but are seeing weaker productivity gains than expected, especially when their code is spread across many repositories. - Engineers evaluating autonomous agents who want a realistic view of their current level of production use. ## Related talks - [Can You Prove AI ROI in Software Engineering?](https://aietalks.com/talks/can-you-prove-ai-roi-in-software-engineering) (Yegor Denisov-Blanch, Stanford, 16:40) - [Developer Experience in the Age of AI Coding Agents](https://aietalks.com/talks/developer-experience-in-the-age-of-ai-coding-agents) (Max Kanat-Alexander, Capital One, 18:20) - [Building an Autonomous Engineering Org](https://aietalks.com/talks/building-an-autonomous-engineering-org) (Angie Jones, Agentic AI Foundation, 17:36) - [AI Leadership](https://aietalks.com/talks/ai-leadership) (Alex Lieberman, Morning Brew and 10X & Kath Korevec, Google Labs & Katelyn Lesse, Anthropic & Michele Catasta, Replit & Lisa Orr, Zapier & Steve Yegge, Sourcegraph and AMP & Gene Kim, IT Revolution & Bill Chen & Brian Fioca, OpenAI & Martin Harrysson & Natasha Maniar, McKinsey & Yegor Denisov-Blanch, Stanford & Itamar Friedman, Qodo & Olive Song, MiniMax & Asaf Bord, Northwestern Mutual & Lei Zhang, Bloomberg & Samir Mody, The Browser Company & Max Kanat-Alexander, Capital One & Arman Hezarkhani, 10X & Justin Reock, DX & Dan Shipper, Every & Mel Lutzky, Graphite, 8:16:05) - [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45)