Better AI Models Won't Fix Your Engineering Team

Nick Arcolano, Jellyfish16:39 · Oct 2026 · 5,087 views
Thumbnail for Better AI Models Won't Fix Your Engineering Team Watch on YouTube
TL;DR
  1. 1

    Switching engineers to Grok produced no measurable change in merged pull requests per engineer, despite being a large and externally driven model change.

  2. 2

    How engineers use AI has a larger effect than model choice. Engineers using skills at least 30 times per week saw about a 30% throughput increase, while token consumption also correlated with higher output.

  3. 3

    The individual engineer explains about three quarters of the variation in the statistical model. Company and team conditions explain about 12%, so leaders should work on people, skills, and engineering environments.

Summary

Nicholas Arcolano examines what drives AI-assisted engineering output using data from about 50,000 engineers at 447 companies. He starts with a golf analogy: better equipment can help, but the player and the course also matter. Jellyfish's analysis found no statistical relationship between a large Cursor-driven shift toward Grok and merged pull requests per engineer. Model and tool choice explained less than 1% of throughput variation. Skill use explained about 3%, token consumption about 9%, and company and team factors about 12%. The individual engineer accounted for roughly three quarters. Skill use had a substantial effect when engineers used it often, while higher token use correlated with more output and diminishing returns. Arcolano also says AI has not greatly reordered engineering performance. Engineers who were top performers before AI are mostly still top performers afterward. His advice is to focus leadership effort on skills, people, teams, and the working environment.

Key ideas
02:01

Model improvements do not automatically produce more engineering output

Arcolano accepts that frontier models have improved and can do things that were not possible a year earlier. He compares this progress with golf clubs, whose technology contributed to higher average PGA Tour driving distances. That comparison does not settle how much models matter in day-to-day engineering work. Jellyfish combines AI coding tool telemetry with engineering workflow and outcome data, allowing Arcolano to compare model changes with merged pull requests. The talk asks whether better models change throughput, rather than assuming that model benchmarks translate directly into engineering results.

02:53

The Grok switch was a natural experiment with no throughput effect

Cursor rapidly increased Grok usage after changes to its interface and auto mode. Arcolano treats this as an external shock rather than a gradual choice by engineers, which makes it useful for comparison. He measures the change in each engineer's model share against the relative change in merged pull requests per engineer, compared with colleagues at the same company. The result clusters around zero. Engineers who moved more of their usage toward Grok did not show a meaningful change in output. Arcolano calls this a non-story and says it left him asking what factors mattered instead.

05:01

Tool and model choice explain less than one percent of throughput variation

For the main analysis, Jellyfish uses data from about 50,000 engineers at 447 companies, focused on Cursor and Claude Code users. Arcolano groups the harness and model together because they are closely connected. He also separates tool choice from skill use, token consumption, workplace conditions, and individual differences. Across the statistical analysis, the tool explains less than 1% of changes in merged pull requests per engineer per week. Switching between Opus and Sonnet, moving to ChatGPT, changing from Cursor to Claude Code, or moving between autocomplete and agentic modes produced a similar result: no meaningful throughput change.

08:07

Using AI skills often has a larger effect than choosing a model

Skill use includes cursor rules, harness customization, and techniques engineers learn for working with AI coding tools. Arcolano says skills matter, but their overall contribution is limited because many engineers rarely use them. In September, almost half of the engineering weeks in his data had essentially no skill use, while only 12% reached a level he considered appropriate for modern AI engineering. Among engineers using skills at least 30 times per week, the factor was associated with a 30% increase in merged PR throughput, holding other factors constant. This effect is much larger than the effect of changing model family.

09:59

More token use correlates with more output, with diminishing returns

Arcolano uses token consumption and active use days to measure how much engineers use AI. His comparison covers about 500,000 developer weeks, grouped by similar weekly consumption. The curve rises as token use increases, then begins to flatten. The top 10% of users can roughly double output compared with the median, but they spend about 10 times as many tokens. Arcolano compares this to rocket fuel: additional consumption produces more output, though each increase brings smaller gains. The result describes a relationship between use and throughput, rather than a claim that spending alone solves engineering problems.

11:23

Company and team conditions create a wide output gap at the same AI spend

Once token consumption is held constant, company and team conditions still explain substantial variation. About one third of this workplace effect comes from the team and two thirds from the company. Arcolano names the code base, review process, and culture as examples. There is a twofold spread between the top 10% and bottom 10% of companies while holding consumption fixed. That means two organizations can spend similar amounts on AI and still produce very different output. This is the part of the analysis that applies most directly to leaders, because they can change parts of the working environment even when they cannot buy more tokens.

12:45

The individual engineer explains most of the measured variation

The individual engineer accounts for about three quarters of the statistical model. The spread between the top 10% and bottom 10% of engineers is about 3.5 times, wider than the twofold company spread. Arcolano connects this result to a question about whether AI requires a different kind of engineer or changes the ranking of engineers. His data does not show a major reshuffling. In a study of 10,000 engineers, about 44% of people in the top quartile changed out of that group over a six-month period before AI, compared with 41% after AI. Engineers who performed best before AI are generally still the strongest performers.

15:07

Leaders can change the engineering environment even when they cannot change the model

Arcolano says model choice gets most of the attention because it is exciting and easy to discuss, but leaders have little control over foundation model progress. Cost limits can also reduce the freedom to buy preferred tools. The factors on the other side of his analysis are more actionable: people, skills, teams, company practices, and the engineering environment. He returns to the golf analogy by saying that leaders can change the course, including conditions that a golfer cannot alter. His closing advice is to question the assumption that a disappointing result comes from the model and examine the team and the individual engineers instead.

"When you get up to 30 plus runs of using skills per engineer per week, that's a 30% bump just from that factor alone holding other things constant."09:35
Who should watch
  • You lead an engineering organization and are deciding whether a new model or coding tool will solve a throughput problem.
  • Your team uses AI coding tools inconsistently and you want evidence about the relative effects of skills, token use, and model choice.
  • You are evaluating whether AI has changed which engineers perform best and want a data-based answer rather than a prediction.