# How to Build a Software Factory That Improves Itself

Zach Lloyd, Warp | AI Engineer NYC 2026 | 22:01

Source: https://www.youtube.com/watch?v=nKjPBAw_N_0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-build-a-software-factory-that-improves-itself
Published: 2026-11-09
Tags: agents, coding-agents, evals, software-factories, workflows

## TL;DR
- Companies can move from engineers interactively prompting agents on laptops to cloud software factories that sandbox agents and coordinate software work.
- A factory should be open to different models and harnesses, work with existing tools, and have its workflows, skills, and infrastructure defined in version-controlled code.
- Stored agent traces can feed automated scoring, factory changes, and benchmarks built from a company's own past tasks.

## Summary
Zach Lloyd describes software factories as cloud systems where humans submit work and agents handle parts of triage, specification, implementation, review, and verification. He argues that local agents are difficult to secure, govern, and control for cost because each engineer configures a separate environment. A cloud factory can centralize those controls while preserving the ability to use different models, harnesses, tools, and compute. Lloyd's main design requirement is that the factory itself be defined in code and stored in a repository. That makes changes testable and lets agents propose updates. Warp's demo shows dashboards for productivity, cycle time, spending, and agent traces. LLM-based scorers find failures such as redundant tests. A self-improvement loop groups those failures and proposes pull requests against the factory. Lloyd also recommends replaying real past tasks to compare model cost and quality on a company's own code. He is direct that the platform is most useful for teams rather than individual developers.

## Key ideas
### Interactive prompting leaves people as the bottleneck
[01:10](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=70s)
Lloyd says most companies are still in an interactive prompting phase. Engineers run tools such as Claude Code or Codex on their laptops and prompt agents to build features or fix bugs. Companies want automated development because they need more business value at lower cost, with governance and security. Local agents create several limits. A compromised agent can access what the signed-in laptop user can access. Separate MCP, skill, and tool configurations make activity hard to see or audit. Lloyd's proposed response is to sandbox agents and move them to the cloud.

### A cloud factory coordinates humans and agents through the software lifecycle
[03:08](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=188s)
Lloyd defines a cloud software factory as a loop where humans submit work and agents handle different parts of the software lifecycle. Agents can triage work, collaborate with a human on a specification, implement changes, and perform computer-use verification. Each company should define how humans and agents interact internally. The factory should then provide ways to measure and improve the process over time. In his framing, the factory affects how an organization ships software because it contains the rules and workflows for this combined human-agent work.

### The factory should be infrastructure defined in version-controlled code
[04:03](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=243s)
Warp's approach treats the factory as an infrastructure stack rather than an AI teammate. Lloyd says it should support different models, including open-weight and frontier models, as well as different harnesses because both change over time. It should adapt to existing infrastructure. Its definition should specify workflows, skills, MCPs, and where components run. Putting that definition in code and version control makes it possible to test factory versions and lets coding agents update the factory. The repository can contain agent definitions, automations, benchmarks, execution locations, skills, and webhooks.

### Companies should own their data, compute, and inference choices
[05:15](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=315s)
Lloyd says AI sovereignty matters to the larger enterprises Warp works with. A company should own the data entering and leaving its factory and should consider running agents on its own infrastructure and compute. He also mentions providing its own inference. The broader goal is to preserve the option to use any model and any harness. His argument is that teams should take ownership of the core ingredients of a factory instead of depending entirely on a fixed external setup.

### Product engineers and factory engineers have different responsibilities
[06:55](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=415s)
In Lloyd's model, product engineers still build customer features and fix bugs, but they do that through the factory by putting work into it. A second role, which he calls factory engineering, improves the system that produces the software. This work includes improving skills, setting context, and changing the model mix. A platform team or general engineers may take on these responsibilities. The distinction matters because the factory itself becomes something the organization has to build, operate, and improve.

### Stored runs provide the data for visibility and improvement
[07:51](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=471s)
The Warp Factories demo shows views for automation, cycle time, and spending. Lloyd says the system records what engineers and automations do, along with the artifacts they create. Managers can inspect individual runs and the conversations behind them. Engineers can submit work through tools they already use, including Slack, Linear, Jira, and GitHub pull request comments. The work is collected in one location, where it can be viewed and stored over time. Those agent traces become raw material for evaluating and changing the factory.

### Automated scores can find recurring failures in agent runs
[12:01](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=721s)
Lloyd describes scores as online evaluations applied to runs passing through the system. Agents inspect the runs and use an LLM as a judge to decide whether they meet a given criterion. One example looks for redundant tests, which Lloyd calls a common coding-agent failure mode. A failed score becomes an opportunity to change the system. People can inspect and grade runs, but Lloyd says agents can do this work more efficiently. The scoring rules are defined in the system and can later be used in benchmarks.

### A self-improvement loop turns failures into factory changes
[13:21](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=801s)
The self-improvement loop aggregates failure modes, searches for patterns, and proposes changes to the factory definition. Lloyd shows an example where a system reports that a failure happened 37 times and suggests a pull request. The proposed fix is to read more positive skills before implementation. This closes a loop from an agent run, through automated evaluation, to a change in the workflow or agent definition. The factory can therefore improve its own process based on problems observed in real work.

### Private task benchmarks give model comparisons that fit the team's code
[14:42](https://www.youtube.com/watch?v=nKjPBAw_N_0&t=882s)
Lloyd says public benchmarks such as SWE-bench and Terminal Bench may not predict how models perform on a company's workflows. A team can add past runs to a benchmark suite, curate tasks by hand, or ask the factory to identify useful examples from stored sessions. The suite can contain representative small, medium, and hard tasks. Replaying those tasks with different model configurations produces a cost-versus-quality comparison. Lloyd's example compares models on correctness and cost, then uses the results to build a routing strategy for the team's own code and stack.

## Notable quotes
- "And so, the solution to both these problems is to sandbox your agents and move them to the cloud." (02:51)
- "The important thing here is that all of this data is getting stored." (09:28)
- "Every one of these failures is an opportunity for improving how your system works." (13:15)
- "The idea behind a self-improvement loop is that agents will go and they'll look at all these failure modes." (13:34)
- "The key thing that I want people to leave with is really set up something that you can measure and improve." (19:22)

## Tools & references mentioned
- Warp
- Warp Factories
- Warp Terminal
- Google Docs
- Claude Code
- Codex
- MCPs
- SWE-bench
- Terminal Bench
- Linear
- Jira
- GitHub
- GLM 53 Flash
- GPT 56

## Who should watch
- You are deciding whether agent work should remain on individual developer laptops or move into a shared, sandboxed cloud system.
- Your team needs to compare models on its own code and tasks rather than rely on public benchmark results.
- You are building an internal platform and need a way to record runs, score failures, and turn recurring problems into changes to the development workflow.

## Editor's note

Zach Lloyd says public benchmarks such as SWE-bench and Terminal Bench may not predict how models perform on a company's workflows. Kitaru records real agent runs as sessions, including model calls and tool results. Replay runs the agent's code against those recorded inputs and responses, so teams can compare model, prompt, or code changes on failures drawn from their own work.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Building your own software factory](https://aietalks.com/talks/building-your-own-software-factory) (Eric Zakariasson, Cursor, 1:23:37)
- [The Software Factory: From Bug Report to Production Code](https://aietalks.com/talks/the-software-factory-from-bug-report-to-production-code) (Davis Palmie, Factory, 18:32)
- [Software Engineering Is Becoming Factory Engineering](https://aietalks.com/talks/software-engineering-is-becoming-factory-engineering) (Zach Lloyd, Warp, 20:37)
- [How Software Factories Improve Themselves](https://aietalks.com/talks/how-software-factories-improve-themselves) (Suraj Gupta, Warp, 12:52)
- [What It Actually Takes to Build a Software Factory](https://aietalks.com/talks/what-it-actually-takes-to-build-a-software-factory) (Tereza Tížková, Factory, 22:49)
