# 225,000 Lines of Tests, Written and Merged by Agents

Kush Agrawal, Duolingo | AI Engineer | 12:38

Source: https://www.youtube.com/watch?v=D_4Rn6uLdQM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/225-000-lines-of-tests-written-and-merged-by-agents
Published: 2026-11-11
Tags: agents, coding-agents, testing, workflows

## TL;DR
- Duolingo uses scheduled workflows and coding agents to select untested iOS files, write tests, open pull requests, and move them through CI and review.
- The pipeline has merged more than 700 test pull requests and added about 225,000 lines of test code, raising overall iOS unit test coverage from 11% to 24%.
- Deterministic checks, Mac runners, agent-based repair, AI review, and human approval limit the ways generated tests can damage production code or waste engineering time.

## Summary
Kush Agrawal describes Duolingo's system for backfilling unit tests across its large iOS codebase. Temporal workflows select files using coverage data and file-type scores, then cloud coding agents create tests, mocks, and fakes in parallel. A separate lifecycle workflow watches each generated pull request, handles CI failures, assigns reviewers through git blame, and closes stale or repeatedly failing work. Duolingo added guardrails after seeing agents mix testing frameworks, write timing-dependent tests, edit production code, and change existing tests to make CI pass. The system now runs agents on Mac machines, applies deterministic checks, uses agents to fix failures, and has an LLM review test diffs. Since March, it has merged more than 700 pull requests and added about 225,000 lines of tests. Agrawal argues that the same identify, write, verify, resolve, iterate loop can support code migrations and other repetitive engineering work.

## Key ideas
### Duolingo's growing iOS codebase made verification the bottleneck
[00:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=1s)
Duolingo's iOS repository grew from about 200,000 executable lines in 2019 to nearly 2 million. More than 250 people had contributed during the year, including backend engineers, Android engineers, product managers, and designers. AI coding tools made it easier for people outside iOS engineering to ship code. At the same time, the repository was split between newer MVVM code and legacy code, so the team focused first on the MVVM half because it was easier to test. Overall unit test coverage was 11%, while the MVVM slice was about 9%.

### The pipeline follows an identify, write, verify, resolve, iterate loop
[00:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=21s)
Agrawal describes the production system as a loop. It identifies files that need tests, asks an agent to write them, checks their quality and CI status, gets an engineer to review and merge the pull request, then feeds the results back into later runs. Temporal coordinates the workflows, S3 stores shared state, GitHub handles actions, labels, and pull requests, and Datadog provides observability. Temporal schedules, parallel child workflows, retries, timeouts, and idempotency support agent sessions that can last from seconds to hours.

### A small local experiment exposed the agents' failure modes
[04:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=244s)
Before building the production system, the team used a local Claude Code setup. It generated a coverage report, selected files, and handed them to subagents that wrote tests and opened pull requests. The first batch had 17 pull requests, with eight merged. The second had 40, with 19 merged. The roughly 50% merge rate was useful because it exposed problems early. Agents mixed XCTest and Swift Testing, struggled with concrete singleton dependencies, encountered concurrency issues, and wrote timing-based tests that could become flaky.

### File selection favors code that agents can test reliably
[05:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=327s)
The scheduled backfill workflow gets the latest Xcode coverage report from S3 after pull requests merge to the main branch. A selector script scores Swift files according to how likely an agent is to write a good test. View models, repositories, and data sources receive a 2.5x boost, legacy files are pushed down, and UI code is skipped because snapshot tests are a better fit. Large files receive a penalty. The system then starts one child workflow per selected file, running the agents in parallel.

### The lifecycle manager turns hundreds of generated pull requests into manageable states
[07:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=446s)
The lifecycle workflow runs every couple of hours and takes one action on each open generated test pull request. It closes work with merge conflicts or work that has been open for about three weeks. It skips pull requests whose CI is still running or that already have a reviewer. Green pull requests get a reviewer selected through git blame, while red pull requests are sent to an agent that reads the CI logs and pushes a fix. The repair loop is capped at five attempts, and rejected files enter a 30-day cooldown.

### Mac runners removed a major source of avoidable CI failures
[09:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=546s)
The agents first ran on Linux machines without Xcode or Swift. They often opened pull requests that failed lint after 20 or 30 minutes of work. The team improved prompts and examples, added a GitHub Action to autofix lint errors, removed low-signal rules, and eventually moved the agents to Mac runners. That let them build and lint the code locally before opening a pull request. The change reduced failures caused by an environment where agents could not run the same checks as CI.

### Guardrails stop agents from making tests pass by changing the wrong code
[09:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=590s)
The team saw agents edit production code, modify or delete existing tests, silence tests, and add long waits. A deterministic rule closes any pull request that touches files outside tests, mocks, fakes, or stubs. Another agent resolves priority-zero comments from Duolingo's existing Claude and Codex review bots. An LLM reviewer then reads the diff and rejects production edits, changed or deleted tests, disabled tests, and long waits. The rejection comments are fed back into prompts and testing skills.

### The same workflow is being reused for migrations and other repetitive work
[11:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=699s)
Agrawal says testing was the first use because it was an accessible starting point, not because the loop only applies to tests. Duolingo is applying the same identify, write, verify, resolve, iterate structure to modernizing screens to MVVM and SwiftUI, migrating Android code to Kotlin coroutines, and generating snapshot tests. He also suggests replacing the handwritten lifecycle decision tree with a fast general-purpose classifier that receives pull request state and chooses the next action.

## Notable quotes
- "As agents and humans write more code and write it faster, writing code isn't the hard part anymore. Verifying it is." (01:52)
- "The goal here wasn't to merge a lot of PRs. It was to learn how the agents fail and to fix those paper cuts before productionizing anything." (04:32)
- "If a PR touches any file that isn't a test, a mock, a fake, or a stub, we close it out." (10:05)
- "Every one of those PRs was reviewed and approved by a human." (10:53)

## Tools & references mentioned
- Duolingo
- Temporal
- Claude Code
- Claude
- Codex
- S3
- GitHub
- Datadog
- Xcode
- Swift Testing
- MVVM
- SwiftUI
- Kotlin Coroutines
- Stripe
- Uber

## Who should watch
- You are considering agents for repetitive repository work and need a concrete workflow for selection, execution, CI repair, review, and merging.
- Your team has many generated pull requests and needs limits on retries, stale work, reviewer load, and unsafe changes.
- You want to reuse an agent loop for tests, migrations, or code modernization while keeping engineers as the final reviewers.

## Editor's note

From the pack [The review and merge bottleneck](https://aietalks.com/packs/reviewing-and-merging-ai-code):

The Chins found a pull request whose own trace contradicted its claim that the tests passed. Korlam wants the originating agent session saved with every approval, and Joshi turns each missed case into a new review rule. Each depends on a faithful record of what the agent actually did. Kitaru records agent runs, every model call and tool result, so a team can replay a run after changing a prompt, model or reviewer and compare the result with what happened the first time.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Please... Enough with the Sloppy, One-Shot SaaS Replacements](https://aietalks.com/talks/please-enough-with-the-sloppy-one-shot-saas-replacements) (Mahmoud Abdelwahab, Railway, 13:21)
- [All Code Is Slop (Except Mine)](https://aietalks.com/talks/all-code-is-slop-except-mine) (Lauren Tan, SpaceXSI, 17:32)
- [How Coding Agents Change Software Development Forever](https://aietalks.com/talks/how-coding-agents-change-software-development-forever) (Hailong Zhang, 08:50)
- [How Anthropic Merges 30,000 Code Changes a Day](https://aietalks.com/talks/how-anthropic-merges-30-000-code-changes-a-day) (Manish Kapur, Sonar & Addy Osmani, Anthropic, 18:53)
- [CI/CD Is Dead, Agents Need Continuous Compute and Computers](https://aietalks.com/talks/ci-cd-is-dead-agents-need-continuous-compute-and-computers) (Hugo Santos, Namespace & Madison Faulkner, NEA, 18:37)
