How to Build a Software Factory That Improves Itself

Zach Lloyd, Warp22:01 · Nov 2026 · 2,118 views
Thumbnail for How to Build a Software Factory That Improves Itself Watch on YouTube
TL;DR
  1. 1

    Companies can move from engineers interactively prompting agents on laptops to cloud software factories that sandbox agents and coordinate software work.

  2. 2

    A factory should be open to different models and harnesses, work with existing tools, and have its workflows, skills, and infrastructure defined in version-controlled code.

  3. 3

    Stored agent traces can feed automated scoring, factory changes, and benchmarks built from a company's own past tasks.

Summary

Zach Lloyd describes software factories as cloud systems where humans submit work and agents handle parts of triage, specification, implementation, review, and verification. He argues that local agents are difficult to secure, govern, and control for cost because each engineer configures a separate environment. A cloud factory can centralize those controls while preserving the ability to use different models, harnesses, tools, and compute. Lloyd's main design requirement is that the factory itself be defined in code and stored in a repository. That makes changes testable and lets agents propose updates. Warp's demo shows dashboards for productivity, cycle time, spending, and agent traces. LLM-based scorers find failures such as redundant tests. A self-improvement loop groups those failures and proposes pull requests against the factory. Lloyd also recommends replaying real past tasks to compare model cost and quality on a company's own code. He is direct that the platform is most useful for teams rather than individual developers.

Key ideas
01:10

Interactive prompting leaves people as the bottleneck

Lloyd says most companies are still in an interactive prompting phase. Engineers run tools such as Claude Code or Codex on their laptops and prompt agents to build features or fix bugs. Companies want automated development because they need more business value at lower cost, with governance and security. Local agents create several limits. A compromised agent can access what the signed-in laptop user can access. Separate MCP, skill, and tool configurations make activity hard to see or audit. Lloyd's proposed response is to sandbox agents and move them to the cloud.

03:08

A cloud factory coordinates humans and agents through the software lifecycle

Lloyd defines a cloud software factory as a loop where humans submit work and agents handle different parts of the software lifecycle. Agents can triage work, collaborate with a human on a specification, implement changes, and perform computer-use verification. Each company should define how humans and agents interact internally. The factory should then provide ways to measure and improve the process over time. In his framing, the factory affects how an organization ships software because it contains the rules and workflows for this combined human-agent work.

04:03

The factory should be infrastructure defined in version-controlled code

Warp's approach treats the factory as an infrastructure stack rather than an AI teammate. Lloyd says it should support different models, including open-weight and frontier models, as well as different harnesses because both change over time. It should adapt to existing infrastructure. Its definition should specify workflows, skills, MCPs, and where components run. Putting that definition in code and version control makes it possible to test factory versions and lets coding agents update the factory. The repository can contain agent definitions, automations, benchmarks, execution locations, skills, and webhooks.

05:15

Companies should own their data, compute, and inference choices

Lloyd says AI sovereignty matters to the larger enterprises Warp works with. A company should own the data entering and leaving its factory and should consider running agents on its own infrastructure and compute. He also mentions providing its own inference. The broader goal is to preserve the option to use any model and any harness. His argument is that teams should take ownership of the core ingredients of a factory instead of depending entirely on a fixed external setup.

06:55

Product engineers and factory engineers have different responsibilities

In Lloyd's model, product engineers still build customer features and fix bugs, but they do that through the factory by putting work into it. A second role, which he calls factory engineering, improves the system that produces the software. This work includes improving skills, setting context, and changing the model mix. A platform team or general engineers may take on these responsibilities. The distinction matters because the factory itself becomes something the organization has to build, operate, and improve.

07:51

Stored runs provide the data for visibility and improvement

The Warp Factories demo shows views for automation, cycle time, and spending. Lloyd says the system records what engineers and automations do, along with the artifacts they create. Managers can inspect individual runs and the conversations behind them. Engineers can submit work through tools they already use, including Slack, Linear, Jira, and GitHub pull request comments. The work is collected in one location, where it can be viewed and stored over time. Those agent traces become raw material for evaluating and changing the factory.

12:01

Automated scores can find recurring failures in agent runs

Lloyd describes scores as online evaluations applied to runs passing through the system. Agents inspect the runs and use an LLM as a judge to decide whether they meet a given criterion. One example looks for redundant tests, which Lloyd calls a common coding-agent failure mode. A failed score becomes an opportunity to change the system. People can inspect and grade runs, but Lloyd says agents can do this work more efficiently. The scoring rules are defined in the system and can later be used in benchmarks.

13:21

A self-improvement loop turns failures into factory changes

The self-improvement loop aggregates failure modes, searches for patterns, and proposes changes to the factory definition. Lloyd shows an example where a system reports that a failure happened 37 times and suggests a pull request. The proposed fix is to read more positive skills before implementation. This closes a loop from an agent run, through automated evaluation, to a change in the workflow or agent definition. The factory can therefore improve its own process based on problems observed in real work.

14:42

Private task benchmarks give model comparisons that fit the team's code

Lloyd says public benchmarks such as SWE-bench and Terminal Bench may not predict how models perform on a company's workflows. A team can add past runs to a benchmark suite, curate tasks by hand, or ask the factory to identify useful examples from stored sessions. The suite can contain representative small, medium, and hard tasks. Replaying those tasks with different model configurations produces a cost-versus-quality comparison. Lloyd's example compares models on correctness and cost, then uses the results to build a routing strategy for the team's own code and stack.

"The idea behind a self-improvement loop is that agents will go and they'll look at all these failure modes."13:34
Who should watch
  • You are deciding whether agent work should remain on individual developer laptops or move into a shared, sandboxed cloud system.
  • Your team needs to compare models on its own code and tasks rather than rely on public benchmark results.
  • You are building an internal platform and need a way to record runs, score failures, and turn recurring problems into changes to the development workflow.