AI adoption produces sharply different results across companies, so average productivity numbers are a poor guide for leadership decisions.
2
Leaders should combine education, time to experiment, psychological safety, and clear policies instead of imposing an adoption quota.
3
AI measurement should connect utilization with speed, quality, and cost, while teams apply AI to bottlenecks across the software development lifecycle.
Summary
Justin Reock argues that AI-assisted engineering needs active leadership rather than a simple tool rollout. Industry averages show modest gains, but company-level data varies widely. Some organizations improve change confidence and quality, while others ship more defects. Leaders should give engineers time to learn, explain why metrics are collected, protect psychological safety, and work with compliance to open safe paths to adoption. Reock recommends measuring utilization alongside established engineering outcomes such as throughput, change failure rate, maintainability, and developer experience. He also urges teams to apply AI beyond code completion, since context switching, incident response, onboarding, and legacy-code analysis can be larger constraints. Examples from Morgan Stanley, Zapier, and Spotify show how AI can target those constraints. The talk presents leadership as the work of setting intent, finding bottlenecks, maintaining shared prompts and rules, and iterating from measured results.
AI productivity results vary sharply between companies
Reock opens with conflicting evidence about generative AI. Google reported that people were 10% more productive, while a METR study found a 19% decrease in productivity with coding assistance. Engineers in that study still felt more productive, which Reock describes as an induced flow. Aggregate DX data showed modest positive changes, including a 2.6% increase in change confidence and a 1% reduction in change failure rate. The company-level view was very different. Some organizations saw 20% increases in change confidence, while others saw 20% decreases. A 2% increase in change failure rate could mean shipping as much as 50% more defects against a 4% industry benchmark.
Top-down adoption quotas create compliance without useful change
Reock says mandates such as 100% AI adoption do not move engineering outcomes. An engineer can update a file every morning simply to appear compliant. Organizations also struggle when they turn on a tool without education or enablement, then expect people to discover good practices on their own. Leaders need clear AI policies, time for engineers to experiment, and a way to measure impact rather than utilization alone. Reock frames these actions as patterns found in organizations that are moving their results in the desired direction.
Psychological safety makes AI rollout a leadership responsibility
Reock connects AI adoption to Google's Project Aristotle, a 2012 study that found psychological safety was the strongest indicator of team productivity. He also cites SWE-bench to show the limits of current agents: they can complete about a third of tasks without human intervention, which leaves two-thirds unfinished. Leaders should communicate that AI is intended to augment engineers and increase business throughput. They should state that intent before fear grows, explain what metrics are being gathered, and help employees build the skills needed to work effectively with the technology.
Speed metrics need quality and developer experience measures beside them
Reock divides the outcomes leaders should watch into speed and quality. Teams may want more pull request throughput and velocity, but faster output can create technical debt. Quality measures include change failure rate, perceived quality, change confidence, and maintainability. API telemetry such as suggestion acceptance provides context, but it can be misleading because the API may not know what engineers rewrite afterward. Reock recommends experience sampling, such as adding an AI-use field to a pull request form, and effective surveys with high participation. He cites W. Edwards Deming's view that 90% to 95% of productivity output is determined by the system rather than the worker.
The AI measurement framework connects utilization to impact and cost
DX's AI Measurement Framework organizes measurement into utilization, impact, and cost. Early measurement asks who uses the technology, what percentage of pull requests are AI-assisted, and how many tasks go to agents. A more mature approach correlates that utilization with velocity and quality. Cost becomes another dimension as teams understand token spending and other usage costs. Reock says AI-specific metrics describe what is happening with the technology, while established developer productivity and experience metrics show whether the initiative is producing the outcomes leaders want.
Reock recommends a gatekeeper for system prompts, Cursor rules, and agent markdown. This person or group receives feedback and maintains the rules over time. He gives the example of models producing outdated Spring Boot 2 guidance when the organization wants Spring Boot 3. The prompt configuration needs a process for correcting that behavior and keeping the assistant aligned with the business. He also explains temperature as a setting that controls randomness. A low value can produce the same output repeatedly, while a high value can produce different valid approaches, so teams should choose settings according to the task.
Education must include time to practise useful AI workflows
DX sampled developers who saved at least an hour a week and asked them to rank their five most valuable use cases. The resulting guide includes code and prompting examples intended to make good practices more reflexive. Reock says some engineering groups have made the guide required reading, but materials alone are insufficient. Leaders also need to give people time to learn and experiment. The top use case in the study was stack trace analysis, which Reock notes is interpretive rather than generative. That result broadens the idea of where AI can help during development.
AI should target the bottleneck across the software development lifecycle
Reock uses the theory of constraints to argue that saving an hour on a non-bottleneck has little value. Data from almost 140,000 engineers showed that AI time savings can be outweighed by context switching, interruptions, and meeting-heavy days. He points to Morgan Stanley's DevGenAI work on legacy code, which creates specifications for modernization and saves about 300,000 hours annually. Zapier uses bots and agents for onboarding and makes new engineers effective in two weeks. Spotify gives site reliability engineers incident context and runbook steps in their channels, reducing time to diagnose incidents. Leaders should find the real constraint before choosing an AI use case.
