# Can You Prove AI ROI in Software Engineering? Yegor Denisov-Blanch, Stanford | AI Engineer CODE 2025 | 16:40 Source: https://www.youtube.com/watch?v=JvosMkuNxF8 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/can-you-prove-ai-roi-in-software-engineering Published: 2025-12-11 Tags: engineering-culture, evals, observability ## TL;DR - AI productivity gains vary widely between teams, and usage volume alone has only a loose relationship with better results. - Cleaner codebases give AI more room to help, while unchecked AI use can increase codebase entropy and reduce engineers' trust. - PR counts can make an AI rollout look successful even when code quality falls, rework rises, and effective output stays flat. ## Summary Yegor Denisov-Blanch presents research on AI-assisted software engineering based on historical and cross-company data. His team trained a machine learning model to reproduce evaluations from panels of software experts who assess implementation time, maintainability, and complexity. The data suggests that AI gains depend more on how teams use the tools and the condition of their codebases than on token volume. Denisov-Blanch proposes measuring tool usage alongside engineering output, with rework, refactoring, quality, technical risk, and people metrics as guardrails. A case study shows why PR counts are misleading: after AI adoption, a large team produced 14% more pull requests, while code quality fell by 9%, rework rose 2.5 times, and effective output did not meaningfully increase. The talk offers a practical way to assess AI ROI without waiting for noisy business outcomes, while admitting that these engineering measures remain proxies. ## Key ideas ### The study measures AI impact with expert-labelled engineering data [00:13](https://www.youtube.com/watch?v=JvosMkuNxF8&t=13s) Denisov-Blanch's research uses both time-series data and cross-sectional data across companies. To measure engineering work at scale, the team collected millions of evaluations from panels of 10 to 15 independent experts. These experts assessed code commits for implementation time, maintainability, and complexity. The team then trained a machine learning model to reproduce the panel's evaluations. This lets them score a much larger body of work, while still allowing a company to assemble its own expert panel and compare the results. ### AI gains are widening between strong and weak adopters [02:01](https://www.youtube.com/watch?v=JvosMkuNxF8&t=121s) The researchers compared 46 teams using AI with 46 similar teams that were not using it, then measured net productivity gains each quarter. The median gain for this cohort was about 10% as of July of the year discussed. The spread between the strongest and weakest teams was growing. Denisov-Blanch uses this as a warning for leaders: successful early adopters may compound their gains, while struggling teams fall further behind. Companies need to measure their own position so they can change course. ### Token volume has only a loose link to productivity [03:07](https://www.youtube.com/watch?v=JvosMkuNxF8&t=187s) The team compared productivity increases with token usage per engineer per month. The relationship was loose, with a linear correlation of about 0.20. The data also showed a possible dip around the 10 million token mark, where teams appeared to perform worse than teams using somewhat fewer tokens. Denisov-Blanch treats this pattern as directional rather than definitive. His practical conclusion is that the quality of AI use matters more than the amount of AI activity. ### Clean codebases give AI more opportunities to help [04:02](https://www.youtube.com/watch?v=JvosMkuNxF8&t=242s) The environment cleanliness index combines tests, types, documentation, modularity, and code quality. The researchers found a correlation of about 0.40 between this index and productivity gains from AI. Denisov-Blanch argues that codebase hygiene creates better conditions for AI-assisted work. He also warns that unchecked AI use can increase technical debt and push cleanliness down. Teams need to maintain the codebase as AI generates more changes. Engineers also need to know when AI is a poor fit, because rejected or heavily rewritten output can erode trust and reduce future use. ### An AI practices benchmark measures how teams actually use tools [06:17](https://www.youtube.com/watch?v=JvosMkuNxF8&t=377s) The research team developed a benchmark that scans a codebase for AI fingerprints, meaning traces of how teams use AI. It measures the share of active engineering work associated with each pattern and repeats the analysis monthly through Git history. The benchmark ranges from level zero, where people do not use AI, through personal use and team-shared prompts, to systems where AI handles specific tasks or orchestrates an entire workflow. The tool is intended to become open source through the SWE-bench research portal. ### Equal access does not produce equal adoption [07:32](https://www.youtube.com/watch?v=JvosMkuNxF8&t=452s) One company in the dataset gave two business units the same AI tools, licenses, spending, and access. Their adoption patterns still differed sharply. One unit used AI for almost 40% of its work, while the other lagged behind. Denisov-Blanch says leaders therefore need to understand how engineers use AI, rather than recording only whether they have access or whether they use it at all. The practices benchmark is meant to expose these differences inside one organization. ### Engineering outcomes are a more usable ROI proxy than business outcomes [08:31](https://www.youtube.com/watch?v=JvosMkuNxF8&t=511s) Business outcomes such as revenue or net revenue retention would be the ideal way to measure AI ROI, but too many other factors sit between an AI rollout and those results. Sales execution, the macroeconomic environment, and product strategy can all confound the comparison. Denisov-Blanch proposes measuring engineering outcomes instead. This assumes that product teams can turn extra engineering capacity into value and that engineering is a meaningful bottleneck. He recommends balanced metrics and a culture that does not weaponize measurement. ### A primary output metric needs quality guardrails [10:28](https://www.youtube.com/watch?v=JvosMkuNxF8&t=628s) The proposed framework has two parts: AI usage and engineering outcomes. Usage can be measured through access-based comparisons or, preferably, telemetry from coding assistants. Vendor data varies in detail, but Git history can also support a retrospective analysis. For outcomes, the primary metric is engineering output from the expert-replication model, rather than lines of code, PR counts, or DORA metrics. Guardrails cover rework and refactoring, quality, technical risk, and people and DevOps measures. The aim is to increase output while keeping these guardrails healthy. ## Notable quotes - "The takeaway here is to invest in codebase hygiene to unlock these AI productivity gains." (04:26) - "Access to AI and even AI usage doesn't mean or doesn't guarantee that AI is going to be used in the same way across a company." (08:13) - "More PRs doesn't mean better." (14:47) - "Effective output didn't increase meaningfully. And then we saw that rework increased by a lot." (14:48) ## Tools & references mentioned - Stanford - SWE-bench research portal - GitHub Copilot - Cursor - DORA ## Who should watch - You are responsible for an enterprise AI coding rollout and need measures that go beyond license counts or assistant activity. - Your team reports more pull requests after adopting AI, but you do not know whether quality, rework, or useful output changed. - You want to compare AI adoption across teams and find out whether codebase hygiene or engineering practices are limiting the gains. ## Related talks - [Does AI Actually Boost Developer Productivity? (100k Devs Study)](https://aietalks.com/talks/does-ai-actually-boost-developer-productivity-100k-devs-study) (Yegor Denisov-Blanch, Stanford University, 18:12) - [The ROI of AI: Why You Need an Eval Framework](https://aietalks.com/talks/the-roi-of-ai-why-you-need-an-eval-framework) (Beyang Liu, Sourcegraph, 25:28) - [Leadership in AI Assisted Engineering](https://aietalks.com/talks/leadership-in-ai-assisted-engineering) (Justin Reock, DX, 18:11) - [What Data from 20m Pull Requests Reveal About AI Transformation](https://aietalks.com/talks/what-data-from-20m-pull-requests-reveal-about-ai-transformation) (Nick Arcolano, Jellyfish, 17:57) - [How AI Is Changing Software Engineering](https://aietalks.com/talks/how-ai-is-changing-software-engineering) (Gergely Orosz, The Pragmatic Engineer, 26:42)