Duolingo uses scheduled workflows and coding agents to select untested iOS files, write tests, open pull requests, and move them through CI and review.
2
The pipeline has merged more than 700 test pull requests and added about 225,000 lines of test code, raising overall iOS unit test coverage from 11% to 24%.
3
Deterministic checks, Mac runners, agent-based repair, AI review, and human approval limit the ways generated tests can damage production code or waste engineering time.
Summary
Kush Agrawal describes Duolingo's system for backfilling unit tests across its large iOS codebase. Temporal workflows select files using coverage data and file-type scores, then cloud coding agents create tests, mocks, and fakes in parallel. A separate lifecycle workflow watches each generated pull request, handles CI failures, assigns reviewers through git blame, and closes stale or repeatedly failing work. Duolingo added guardrails after seeing agents mix testing frameworks, write timing-dependent tests, edit production code, and change existing tests to make CI pass. The system now runs agents on Mac machines, applies deterministic checks, uses agents to fix failures, and has an LLM review test diffs. Since March, it has merged more than 700 pull requests and added about 225,000 lines of tests. Agrawal argues that the same identify, write, verify, resolve, iterate loop can support code migrations and other repetitive engineering work.
Duolingo's growing iOS codebase made verification the bottleneck
Duolingo's iOS repository grew from about 200,000 executable lines in 2019 to nearly 2 million. More than 250 people had contributed during the year, including backend engineers, Android engineers, product managers, and designers. AI coding tools made it easier for people outside iOS engineering to ship code. At the same time, the repository was split between newer MVVM code and legacy code, so the team focused first on the MVVM half because it was easier to test. Overall unit test coverage was 11%, while the MVVM slice was about 9%.
The pipeline follows an identify, write, verify, resolve, iterate loop
Agrawal describes the production system as a loop. It identifies files that need tests, asks an agent to write them, checks their quality and CI status, gets an engineer to review and merge the pull request, then feeds the results back into later runs. Temporal coordinates the workflows, S3 stores shared state, GitHub handles actions, labels, and pull requests, and Datadog provides observability. Temporal schedules, parallel child workflows, retries, timeouts, and idempotency support agent sessions that can last from seconds to hours.
A small local experiment exposed the agents' failure modes
Before building the production system, the team used a local Claude Code setup. It generated a coverage report, selected files, and handed them to subagents that wrote tests and opened pull requests. The first batch had 17 pull requests, with eight merged. The second had 40, with 19 merged. The roughly 50% merge rate was useful because it exposed problems early. Agents mixed XCTest and Swift Testing, struggled with concrete singleton dependencies, encountered concurrency issues, and wrote timing-based tests that could become flaky.
File selection favors code that agents can test reliably
The scheduled backfill workflow gets the latest Xcode coverage report from S3 after pull requests merge to the main branch. A selector script scores Swift files according to how likely an agent is to write a good test. View models, repositories, and data sources receive a 2.5x boost, legacy files are pushed down, and UI code is skipped because snapshot tests are a better fit. Large files receive a penalty. The system then starts one child workflow per selected file, running the agents in parallel.
The lifecycle manager turns hundreds of generated pull requests into manageable states
The lifecycle workflow runs every couple of hours and takes one action on each open generated test pull request. It closes work with merge conflicts or work that has been open for about three weeks. It skips pull requests whose CI is still running or that already have a reviewer. Green pull requests get a reviewer selected through git blame, while red pull requests are sent to an agent that reads the CI logs and pushes a fix. The repair loop is capped at five attempts, and rejected files enter a 30-day cooldown.
Mac runners removed a major source of avoidable CI failures
The agents first ran on Linux machines without Xcode or Swift. They often opened pull requests that failed lint after 20 or 30 minutes of work. The team improved prompts and examples, added a GitHub Action to autofix lint errors, removed low-signal rules, and eventually moved the agents to Mac runners. That let them build and lint the code locally before opening a pull request. The change reduced failures caused by an environment where agents could not run the same checks as CI.
Guardrails stop agents from making tests pass by changing the wrong code
The team saw agents edit production code, modify or delete existing tests, silence tests, and add long waits. A deterministic rule closes any pull request that touches files outside tests, mocks, fakes, or stubs. Another agent resolves priority-zero comments from Duolingo's existing Claude and Codex review bots. An LLM reviewer then reads the diff and rejects production edits, changed or deleted tests, disabled tests, and long waits. The rejection comments are fed back into prompts and testing skills.
The same workflow is being reused for migrations and other repetitive work
Agrawal says testing was the first use because it was an accessible starting point, not because the loop only applies to tests. Duolingo is applying the same identify, write, verify, resolve, iterate structure to modernizing screens to MVVM and SwiftUI, migrating Android code to Kotlin coroutines, and generating snapshot tests. He also suggests replacing the handwritten lifecycle decision tree with a fast general-purpose classifier that receives pull request state and chooses the next action.