AI coding tools are difficult to evaluate because their value often appears in developer focus, completed work, and progress on important projects rather than a simple output count.
2
Teams can compare tool and non-tool groups, estimate time saved from product actions, track organization-specific outcomes, or ask developers which tool helped most within a fixed budget.
3
No evaluation method is complete, so leaders should define success criteria before a pilot and judge AI tools against the work their organization actually values.
Summary
Beyang Liu argues that measuring AI ROI in software development is difficult because it depends on measuring developer productivity, which has no settled formula. He presents several approaches used by Sourcegraph customers. A controlled comparison can measure whether a team with the tool moves faster, while product engagement data can provide a conservative estimate of time saved. Other companies track specific outcomes, such as feature building, unit testing, code review, or progress on a major migration. Developer surveys are less rigorous, but useful when teams choose within a fixed productivity-tool budget. Liu rejects lines of code as a useful measure and says no framework captures every benefit, including work that would otherwise remain undone. He closes by predicting a move from inline completion toward editor-based agents that can respond to human guidance, while code remains important because it expresses requirements precisely.
AI helps developers stay in flow by closing the gaps around a coding task
Liu describes the ordinary development experience as a series of side quests. A task may look straightforward, then require installing dependencies, learning a framework, or finding an unfamiliar UI component. Without AI, filling one of these gaps can take 30 minutes and two cups of coffee, after which the developer may have forgotten the original task. Cody uses Sourcegraph's broader code context to help answer questions and generate code. Liu says the benefit is that developers can keep their attention on the feature or user experience they are trying to build, instead of abandoning the work when the surrounding details become difficult.
AI ROI is hard because it reduces to the unresolved problem of measuring developer productivity
Liu uses a bean-business analogy to describe the disagreement inside a company. Developers grow the beans, sales sells them, and finance counts them. After AI arrives, finance asks how many more beans engineering produces and whether the tool cost is justified. Liu calls this problem 'NP hard' as a joke about its difficulty: measuring AI ROI means measuring the productivity of knowledge workers, and there is no agreed way to measure developer productivity precisely. He does not conclude that evaluation is impossible. Instead, he recommends practical methods that give organizations useful evidence even when no method captures the full value.
Comparing teams with and without the tool gives a rigorous but expensive estimate
In the A/B testing approach, one group receives the AI tool and another does not. Both groups estimate the work involved in features, bugs, and other planned tasks, then leaders compare execution against those estimates. Palo Alto Networks used a similar method and concluded that approximate timelines accelerated by 20 to 30 percent with Cody. Liu warns that teams differ in architecture, responsibilities, staffing, and unexpected events, so the comparison needs to account for confounding factors. Running the test also requires enough time and planning to build confidence. When a company has those resources, he considers this a strong way to test whether a tool works.
Product engagement can produce a conservative lower bound for time saved
This method starts with product analytics. A team identifies actions that plausibly save time, such as code searches, chats, codebase questions, and inline generation, then assigns each action an approximate time-saving value. Leaders can lower-bound the estimate, such as assuming a search saves two minutes even when some searches prevent half a day or a week of unproductive work. Cody provides administrators with fine-grained usage data for these actions. Liu warns that raw engagement is misleading for autocomplete because the user does not explicitly choose every suggestion. Acceptance rate is a better measure there. The approach makes a tool's minimum economic value easy to explain, but misses work that AI makes possible and broader effects on major initiatives.
Teams should measure the outcomes they want instead of generic activity counts
Liu rejects lines of code as a productivity measure, even though the metric has returned in discussions about AI ROI. A product that triggers more aggressively can generate more lines without creating better work. He recommends that organizations choose measures tied to changes they actually want. A government contractor working with Sourcegraph identified time spent answering questions and interrupting teammates as costs to reduce. It also wanted more developer time spent building features, writing unit tests, and reviewing code. Those activities became the measures for its evaluation period. The example shows why a useful metric depends on the organization's goals and the type of work its engineers do.
The value of an AI tool can be tied to progress on a major engineering initiative
Some customers evaluate AI against a migration or another large project that already matters to engineering leadership. Examples include moving from COBOL to Java or changing a frontend framework. The calculation asks what it is worth to bring the initiative forward by months, years, or to make it possible at all. Liu admits that estimating the amount of time pulled forward involves judgment from the engineering leader. Leaders may already understand the project's pain and can estimate how the product changes the work. Sourcegraph also uses case studies where work that had taken a year or longer was completed in a few months, though Liu presents this as evidence for a specific project rather than a universal benchmark.
Developer surveys are useful when choice is constrained by a fixed budget
A survey is less rigorous than a controlled test, but Liu still considers it valuable. An organization can run a pilot, let developers compare tools, and ask which one worked best. He says this is now often done inside a defined developer-productivity budget rather than through an unrestricted search for whatever makes developers happy. The talk cites a Pragmatic Engineer newsletter survey showing that organizations commonly allocate somewhere between 5 and 25 percent of their budget in the relevant chart. Within that constraint, leaders can ask developers which tools they want most. The result reflects user preference and practical usefulness, but it does not establish a precise productivity gain.
AI development is moving from inline completion toward agents that respond to human guidance
Liu describes a progression from basic inline completion to offline automation. He expects the next stage to be online agents that live in the editor and can react to human feedback and guidance. His argument is that teams should improve reliability and generality step by step rather than jump directly to full automation, which he calls a difficult problem. He also asks whether AI changes the lessons from The Mythical Man-Month. In his view, a highly productive engineer may be more valuable than many mediocre developers who need extensive guidance. AI could become a large multiplier for strong human developers, while code remains the precise output that records what software does.
"AI helps Bridge those gaps it helps uh developers really stay in flow and stay uh kind of like cognizant of the high level of what they're trying to accomplish"05:04
Who should watch
You are an engineering or finance leader deciding whether an AI coding tool has paid for itself.
Your organization is planning a pilot and needs a measurement method that matches its engineering work.
You are tired of lines of code and vague developer sentiment being treated as proof of AI value.
