# Code Generation and Maintenance at Scale

Morgante Pell, Grit | AI Engineer World's Fair 2024 | 18:54

Source: https://www.youtube.com/watch?v=Ve-akpov78Q
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/code-generation-and-maintenance-at-scale
Published: 2024-10-17
Tags: agents, code-generation, coding-agents, harness-engineering, reliability

## TL;DR
- Large-scale software work is mostly about changing existing applications, so AI tools need to operate across repositories and files rather than only autocomplete individual lines.
- Reliable coding agents need static analysis, compilers, language servers, checkpoints, and evaluation loops alongside language models.
- Grit makes large migrations practical by having one engineer coordinate agents that search, edit, validate, and submit pull requests across many repositories.

## Summary
Morgante Pell argues that software development tools should focus more on maintaining existing code than generating new applications. At Google, customers usually wanted to run established business systems on Kubernetes, and he saw the same pattern in other development work. Grit therefore targets senior engineers who can direct agents across large codebases. Pell explains how Grit combines semantic search with static analysis, dependency graphs, compiler feedback, language servers, virtual machine snapshots, and parallel agents. A migration from logging to OpenTelemetry illustrates the approach. The system finds relevant code with structured queries, asks agents to make edits, validates them, and returns pull requests for review. Pell is candid about failure modes. Repeated prompts compound errors, large builds are slow, and models waste tokens regenerating whole files. Checkpoints, parallel branches, and loose search-and-replace edits make the workflow faster and more reliable. He ends with a vision for a codebase interface that can zoom from system-wide structure to individual changes.

## Key ideas
### Most developer work changes large existing applications
[00:47](https://www.youtube.com/watch?v=Ve-akpov78Q&t=47s)
Morgante Pell says his Google Cloud customers rarely asked for a brand-new application. About 90% came with an established line-of-business system and asked how to run it on Kubernetes. The automation tools his teams built focused on those existing systems. He says developers spend most of their time modifying huge applications so that real businesses keep operating. This is why he is skeptical of demos that only show a prompt producing a new app from scratch. Grit is aimed at maintenance and modernization work, where changes may span hundreds or thousands of repositories.

### Grit raises the ceiling for senior engineers
[01:50](https://www.youtube.com/watch?v=Ve-akpov78Q&t=110s)
Pell separates developer assistance into autocomplete in the IDE, agents that lower the skill floor, and tools that raise the ceiling for experienced engineers. He is skeptical that most good software will be built by handing specifications to nontechnical users, because edge cases and careful system design still require engineering judgment. Principal engineers are mainly limited by time and cannot work in ten places at once. Grit lets one engineer direct agents across many repositories. In a typical customer team, one deeply embedded engineer may generate hundreds of pull requests while most other engineers only review the changes that arrive in their repositories.

### Large migrations are coordination problems as much as coding problems
[03:19](https://www.youtube.com/watch?v=Ve-akpov78Q&t=199s)
Pell uses an OpenTelemetry migration as an example. Replacing logging across thousands of repositories normally requires coordination among hundreds of teams, education, code changes, and a program manager tracking progress in a spreadsheet. He says Grit competes with that spreadsheet because the difficult part is often organizing the work. One engineer can coordinate agents that make the changes, open pull requests, fix problems, and merge the results. The customer had postponed the project for years, then completed it in a week with fewer than 100 developer hours, while the agents used thousands of compute hours.

### Reliable agents need code structure, not only embeddings
[05:22](https://www.youtube.com/watch?v=Ve-akpov78Q&t=322s)
Grit first indexes the whole codebase in two ways. Semantic indexing captures the intent of files, while static analysis captures code structure, imports, and dependency relationships. Pell says a simple RAG system that chunks files and retrieves the closest embeddings will fail for the logging migration. It may find user-facing alerts that resemble logs, and it cannot know whether the task requires ten matches or 10,000. Grit's GQL query engine combines syntactic queries, semantic similarity, and import-graph checks. A query can find function calls whose arguments look like error messages and verify that the logging library is actually Log4j.

### Compiler feedback corrects mistakes that humans and models miss
[08:48](https://www.youtube.com/watch?v=Ve-akpov78Q&t=528s)
Pell demonstrates a TypeScript function generated by Claude 3.5 Sonnet. The code looks reasonable, and the full context already contains the information needed to write it, but the function uses incompatible range types and would break the VS Code extension. Asking the model to inspect the code again does not fix the problem because it is not grounded in the actual error. The TypeScript compiler reports that Grit positions and ranges differ from LSP ranges. Feeding that compiler output back to the model produces the correct conversion. Pell calls the loop prompt, build, type-check, and repair roughly half of what a good agent needs.

### Enterprise build times can dominate model latency
[11:19](https://www.youtube.com/watch?v=Ve-akpov78Q&t=679s)
For one customer, building the application from scratch takes ten minutes just for type checking. The model takes about 30 seconds to generate a change, so the build time dominates the agent loop. Repeating that process for every edit can make a simple change take a day. Pell compares this with how developers work in an IDE. Language servers prepare an in-memory index and recheck only the affected parts. With a TypeScript server, the loop can look like 30 seconds for generation, one second for recomputation, and 30 seconds for repair. Agents need these incremental tools instead of relying only on slow command-line builds.

### Checkpoints and parallel branches limit compounding failures
[13:10](https://www.youtube.com/watch?v=Ve-akpov78Q&t=790s)
Agents can get stuck repeatedly fixing the same error, and each failed attempt pollutes the context with irrelevant information. Pell says success falls sharply after more than ten prompts in a row. Grit saves a known-good state and returns to it when a branch goes wrong. Using Firecracker, the system snapshots an in-memory development environment and forks it into as many as ten isolated environments. Agents try different edits in parallel. Grit evaluates them with language-model checks and simpler measures such as error counts and passing unit tests, then selects a good state to continue from. A pull request may involve 30 or 40 generations overall, while the final successful chain contains only four.

### Edit formats affect both cost and accuracy
[15:35](https://www.youtube.com/watch?v=Ve-akpov78Q&t=935s)
Regenerating an entire file is easy to implement, but it costs more and encourages models to leave unchanged code untouched. Pell notes that output tokens are more expensive than input tokens, and model output limits have not grown as quickly as context windows. Unified diffs save output but give models difficult line-number work and are unlike the full files in their training data. JSON-based search and replace wastes tokens escaping code. Grit instead uses a loose search-and-replace format through GQL: the model supplies a before snippet and an after snippet, and the system finds the closest matching code before applying the replacement.

## Notable quotes
- "Developers spend most of their time modifying huge applications so that flights run on time." (01:37)
- "Grit we compete with is actually Excel." (04:23)
- "Compilers rock." (11:00)
- "You can end up with an AI system that looks more like a distributed database than it does a traditional agent or something that you're running on your laptop." (14:43)

## Tools & references mentioned
- Grit
- Google Cloud
- Kubernetes
- Customize
- Terraform
- OpenTelemetry
- GQL
- Claude 3.5 Sonnet
- TypeScript
- VS Code
- TS Server
- Firecracker
- AWS Lima
- Log4j
- LSP

## Who should watch
- You maintain a large codebase and need to coordinate migrations across many repositories or teams.
- You are building coding agents and need practical patterns for search, validation, incremental builds, and recovery from bad branches.
- Your current agent loop is slow or unreliable because it regenerates files, waits on full builds, or repeats failed fixes.

## Related talks

- [How Coding Agents Change Software Development Forever](https://aietalks.com/talks/how-coding-agents-change-software-development-forever) (Hailong Zhang, 08:50)
- [Making Codebases Agent Ready](https://aietalks.com/talks/making-codebases-agent-ready) (Eno Reyes, Factory AI, 15:33)
- [Beyond the Prototype: Using AI to Write High-Quality Code](https://aietalks.com/talks/beyond-the-prototype-using-ai-to-write-high-quality-code) (Josh Albrecht, Imbue, 17:59)
- [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45)
- [Developer Experience in the Age of AI Coding Agents](https://aietalks.com/talks/developer-experience-in-the-age-of-ai-coding-agents) (Max Kanat-Alexander, Capital One, 18:20)