The Code AI Maturity Model and What It Means For You

Ado Kukic, Sourcegraph07:58 · Feb 2024 · 1,946 views
Thumbnail for The Code AI Maturity Model and What It Means For You Watch on YouTube
TL;DR
  1. 1

    The Code AI Maturity Model has six levels, moving from manually written code to AI handling the full software development lifecycle.

  2. 2

    Levels zero through two keep the developer as the primary coder, while the AI improves from basic completion to codebase-specific suggestions.

  3. 3

    At levels four and five, AI can act proactively, submit fixes for review, and eventually take software from requirements to production with minimal human guidance.

Summary

Ado Kukic presents Sourcegraph's six-level Code AI Maturity Model for describing how coding assistants progress from simple autocomplete to autonomous software development. The model has three stages: human-initiated work, AI-initiated work, and AI-led code. Level zero has no AI code generation. Level one generates lines or blocks from developer intent, while level two uses specific context from the current codebase. Level three turns high-level requirements into complete components and integrations. At level four, an assistant can monitor changes or bug reports and submit pull requests without being asked. Level five covers the full lifecycle, including architecture, production code, deployment, and ongoing improvements. Kukic compares these levels with the SAE vehicle autonomy scale, using the comparison to explain changing responsibility between humans and AI. He also connects the model to Cody, Sourcegraph's coding assistant, and invites developers to debate whether the six levels are useful.

Key ideas
00:37

The model separates coding assistance into three kinds of responsibility

Sourcegraph uses six levels of Code AI to describe how coding assistants change over time. The levels fall into three categories. In the human-initiated category, humans remain the primary coders. In the AI-initiated category, AI starts taking a proactive role in software development. In the AI-led category, AI has full autonomy over a codebase. Kukic compares these categories with the SAE levels used for vehicle automation. The comparison gives the model a way to describe who is responsible for driving software work at each stage, from a developer writing everything manually to an AI system taking code from a concept through production.

01:57

Level zero is traditional development without AI-generated code

At level zero, the developer writes all code manually and is responsible for writing, testing, and debugging the codebase. AI does not generate or modify code. Ordinary IDE features such as symbol name completion can still provide limited assistance, but they do not change who creates the software. Kukic describes this as the traditional software development process before AI assistance enters the workflow. His vehicle comparison is a fully human-driven car, where the driver controls acceleration, steering, braking, and everything else. The important boundary is that the AI does not produce any part of the codebase.

02:18

Level one adds code completion based on developer intent

At level one, an AI assistant generates a single line or a larger block of code after the developer provides an indication of what they want. Kukic gives the example of a developer writing a function signature and the assistant inferring the implementation. The assistant has been trained on millions of lines of open source code and uses that material to produce completions based on the developer's guidance. The developer still directs the work and writes the surrounding code. Kukic compares this stage with SAE level one driving, where features such as cruise control or lane centering make driving easier while the human remains fully responsible for the vehicle.

03:06

Level two makes suggestions specific to the current codebase

A level two assistant has a better understanding of the codebase it is working in. Level one uses broad, general context, while level two uses specific information about the project's code and libraries. Kukic uses a Node.js application with the Axios library as an example. A level two assistant should suggest completions based on Axios rather than a different Node HTTP library such as Fetch or SuperAgent. The assistant can therefore produce suggestions that fit the project's actual dependencies and patterns. The developer remains in control, similar to an SAE level two vehicle, where automation can handle some tasks but the human can override it at any time.

04:13

Level three turns high-level requirements into complete software components

At level three, the developer gives a high-level requirement and the assistant delivers a code-based solution. The work goes beyond isolated snippets to include full components and integrations with other software. Kukic's example is asking an assistant to add user authentication to an application. The assistant would generate all the code required, then explain what it wrote, how it works, and how it connects to the rest of the application. The developer still initiates the task and reviews the result. Kukic compares this with SAE level three, where the vehicle takes the primary role in driving but must return control to a human when it cannot safely handle a situation.

05:12

Level four lets an assistant initiate and submit coding work

At level four, the assistant can handle coding tasks proactively without direct developer oversight. Kukic describes an assistant that continuously monitors code changes and submits pull requests to keep documentation current. It could also monitor customer bug reports and submit pull requests that fix the reported issues. The human developer would review the pull requests and merge them. This changes the workflow because the developer does not have to ask for each change before the assistant starts. Kukic compares the stage with SAE level four vehicles, which can perform virtually all driving tasks under specific conditions. He cites Waymo's automated taxis as an example of vehicles operating without a human driver in mapped cities.

06:17

Level five gives AI responsibility for the complete software lifecycle

At level five, the assistant needs minimal human guidance for code generation and can handle the entire software development lifecycle. The developer supplies high-level requirements and specifications. The AI designs the architecture, writes production-quality code, handles deployment, and continuously improves the codebase. The developer validates that the final product meets the stated requirements but does not necessarily inspect the generated code. In Kukic's model, the assistant has complete autonomy from concept to production. The comparison is an SAE level five vehicle, which can perform all driving tasks under all conditions, with humans optional and no need for a steering wheel or human override.

"At level two the AI coding assistant has superior understanding and context of the codebase it is interacting with."03:06
Who should watch
  • You are evaluating coding assistants and need a vocabulary for comparing autocomplete, codebase-aware tools, and more autonomous systems.
  • Your team is deciding how much responsibility an AI coding assistant should have for implementation, pull requests, deployment, or review.
  • You are building or using developer tools and want to think through the boundaries between human-initiated and AI-initiated work.