A better harness cannot fix coding models that are trained to pass tests without preserving maintainability.
2
Turning off code review failed for HumanLayer when an issue appeared that the agent could not solve and the codebase had become difficult to understand.
3
Product review, architecture, program design, and vertical slices let teams keep reading the code while reducing review time.
Summary
Dex Horthy argues that software factories fail when they remove humans from code review before models can preserve a codebase's quality. Coding models are trained mainly through rewards for correctness and passing tests. Those rewards do not capture poor program design, tangled dependencies, or the cost of changes that become difficult months later. Horthy describes HumanLayer's July 2025 lights-off experiment, where nobody read the code. When a serious issue appeared, the team had to return to a codebase it had stopped reading, while the site was down and users were unhappy. His practical response is to turn the lights back on and plan before implementation. Product review, system architecture, program design down to types and call graphs, and vertical slices reduce rework and make PRs easier to review. AI can speed this planning and implementation, but engineers still own the code.
The promise of lights-out factories leaves review as the bottleneck
The prevailing story says engineers are the bottleneck, models are good enough, and teams should spend more tokens and ship more code. Horthy describes an agentic factory that replaces the human implementation step with an agent, then adds orchestration, harnesses, sandboxes, automated review, and regression testing. Implementation may take minutes or hours, but human review and testing still take hours or days. The lights-off version removes code reading altogether and invests in testing, monitoring, and rollout instead. Horthy argues that this approach simply increases the amount of code entering the system without solving the quality problem.
The failure is a model-training problem rather than a prompting problem
Horthy rejects the idea that teams can solve this through better prompting, more loops, or more elaborate harness engineering. He says the underlying problem is how coding models are trained. The usual reward asks whether the requested behavior works and whether existing tests still pass. It does not penalize poor architecture or code that makes future changes harder. A harness can organize the model's work and add review agents, but it cannot teach a model qualities that are absent from its training signal. Horthy is direct that engineers should keep reading code for now, even though that limits the fantasy of a fully autonomous factory.
HumanLayer's July 2025 lights-off experiment exposed the limits
HumanLayer tried running a fully lights-off factory in July 2025. Horthy says the team eventually met an issue that the agent could not solve, even after advanced prompting, research, and reproductions. He had to dig through a codebase he had stopped reading three months earlier. During the incident, the site was down and users were unhappy, while the team faced code it had allowed to become difficult to understand. He says this was not specific to a small side project. Agents began struggling with complex brownfield systems after only three to six months, especially when teams were shipping quickly.
Coding agents optimize for passing tests, which leaves architecture outside the reward
Horthy explains the problem through common coding-agent training tasks. A model receives a software issue, generates attempts, and gets rewarded when the old tests and a hidden new test pass without breaking existing behavior. The process removes changes to test files before evaluation, yet it still has no way to penalize unnecessary try-catch blocks, awkward casts, or designs that erode maintainability. He compares maintainability problems to Martin Fowler's shotgun surgery, where changing one part of a system can break other parts. The cost of bad architecture appears months or years later, so it is hard to send that signal back into reinforcement learning.
Claude Code benefited from training against its own harness
Horthy attributes Claude Code's rapid adoption to a difference in training, rather than to a unique set of basic tools. Earlier CLI agents such as Aider and Code Buff had read, write, edit, grep, and bash. Claude Code was the first example he gives of a model lab training a model against the harness it would distribute to users. He also cites an OpenAI talk arguing that a harness builder without access to model weights, and without the ability to reinforce the model inside its harness, is at a disadvantage against a company that owns both. The model became especially good at calling tools in an agentic loop.
Existing benchmarks measure correctness more easily than maintainability
Horthy says there are no good benchmarks for a model's ability to maintain codebase quality over time. He mentions longer tasks and newer evaluation designs as steps toward better measurement. Sweep Marathon uses very large tasks such as cloning the features of Microsoft Excel. Deep Sweep evaluates large tasks on open-source repositories that were not in the training set. Frontier Code evaluates multi-PR work, penalizes tests that do not fail on the pre-patch code, and uses a judge model for code-quality rules. Horthy doubts judge models can solve the whole problem. If a model already knew what good code looked like, he says, it would probably write it in the first place.
Planning before implementation makes human review affordable
Horthy's practical answer is to turn the lights back on and plan up front. Small changes can still go straight to the agent. Larger work starts with product review, including the problem, desired behavior, and mock-ups. System architecture then defines component contracts, data models, constraints, and how the pieces fit together. Program design goes further into types, method signatures, program layout, and call graphs. Vertical slices define implementation order, coordination across repositories, and checks along the way. Horthy says thirty minutes of pre-planning and alignment can save hours of review, making it feasible to read every line of generated code.
Horthy says teams drowning in pull requests usually have too many bad pull requests, rather than simply too many pull requests. A good PR is easy to compare with the plan: the reviewer can read it and see that it matches what was discussed. A PR needing twenty percent rework creates an emotional and intellectual burden for both the reviewer and the person who submitted it. AI-assisted planning can shorten alignment by gathering information early, make coding faster, and reduce review time because the team agreed on the design first. Engineers still read the code and remain responsible for it.
"There's no way in this system that we can penalize it for poor program design or for eroding the maintainability of our systems."12:47
Who should watch
You are building an agent-driven workflow for a complex or long-lived codebase and are considering removing human code review.
Your team is shipping AI-generated changes quickly, but outages, rising bugs, or weaker PR reviews are making the speed harder to trust.
You want a concrete planning process that uses AI while keeping engineers responsible for the resulting code.
