Long-running agents need a changing harness that addresses context, planning, and judgment as models improve.
2
A separate adversarial evaluator produces better results than asking a generator to judge its own work.
3
Structured handoffs, testable contracts, persistent files, and trace reading help agents work coherently for hours.
Summary
Ash Prabaker and Andrew Wilson describe how Anthropic has built agents that can work for hours or days. They separate improvements in model capability from improvements in the harness around the model. The talk covers context management, planning, checkpoints, sub-agents, skills, compaction, and agent teams. Ash then presents a planner, generator, and adversarial evaluator pattern. The planner turns a broad request into high-level sprints. The generator and evaluator negotiate a specific contract before implementation. The evaluator uses the running application, rather than only reading code or diffs, and sends concrete critiques back to the generator. Rubrics make subjective qualities such as visual design more gradable. The speakers are candid that this approach is expensive and especially suited to greenfield applications. They also describe how harnesses should change with each model release. Some loops become unnecessary as models improve. Reading traces remains the main way to find where model judgment diverges from the intended behavior.
Long-running agents fail through context, planning, and self-judgment
Andrew Wilson groups the problem into three areas. A context window is finite, and a new session creates amnesia, so the agent needs memory components. Coherence also declines deeper into a session, which he calls context rot. Near the end of a window, a model can show context sense anxiety and hurry to finish. Planning is another weakness: a model may attempt everything in one shot, stop after half a feature, or run out of context with an unfinished application. Models also judge their own output poorly. A button may appear complete even though its backend does not exist, and the model may move on anyway.
Model releases and harness changes improve agents together
The speakers describe two ways to extend agent runs. Models can improve through their weights, while the harness can add scaffolding around the model. Andrew gives a minimal-scaffold comparison that moves from about one hour with Opus 3.7 to about twelve hours with Opus 4.6. The harness includes the agent loop, tools, MCP servers, sub-agents, context files, skills, slash commands, and permissions. Andrew says these pieces co-evolve. A harness can fill a model's current gap, and later model training may make part of that harness unnecessary. The harness changes with each model release instead of disappearing when models improve.
Fresh sessions, artifacts, and verification created the first long-running pattern
The first long-running harness began with an initializer agent that converted a vague request such as 'create a Slack clone' into persistent artifacts. It created a feature list, a progress file, a Git repository, an initialization script, and completion flags. The main loop started each task in a fresh context, checked the working directory and progress file, ran a smoke test, selected one unfinished feature, implemented it, and verified it with the running application. A passing feature was committed and marked complete. Andrew says JSON files were less likely than Markdown files to be overwritten by the model. The pattern combined planning, persistent state, fresh contexts, and a verification loop.
A separate evaluator is better at criticism than a generator judging itself
Ash Prabaker compares the generator-evaluator setup to a generative adversarial network. The generator builds, while a separate evaluator grades. The evaluator opens live pages with Playwright, clicks around, tries actions, and returns a critique. This differs from telling one Claude Code session to check its own work. Ash argues that tuning a standalone critic to be harsh is more tractable than making a builder self-critical. The evaluator can still prefer language-model-style output, but separating its context, system prompt, and job creates adversarial pressure. If a generator repeatedly fails one criterion, the harness can discard the work and start again instead of endlessly patching the same approach.
For full-stack applications, correctness is only one part of quality. Ash also considers visual appeal, feel, and taste. His rubric uses design, originality, craft, and functionality, with extra weight on design and originality because Opus 4.6 already performs well on functionality. Few-shot examples of reference sites calibrate the evaluator's preferences and help it avoid generic AI aesthetics such as purple gradients. The evaluator launches the application, takes screenshots, scores the four criteria, writes a critique, and hands it back to the generator. Ash says that a strong opinion about what good output looks like can be written down and graded.
Planner, generator, and evaluator negotiate sprint contracts
The planner takes a one-line request and turns it into a high-level sequence of sprints. It avoids specifying every technical detail because an early mistake could cascade through every later sprint. Before implementation, the generator proposes what it will build and how the evaluator should test it. The evaluator can reject an overly broad scope, strengthen weak tests, or add a missing edge case. They exchange files on disk until they agree. The evaluator then grades against this negotiated contract rather than the planner's original broad specification. Ash describes this as a bridge from user stories to tangible assertions. The planner sets the outer boundaries, while the builder and evaluator decide the exact feature and test details.
In the Retro Forge example, the agents settled on 27 contract criteria. Ash says vague criteria produce vague critiques, which the generator can ignore. Granular criteria tell it exactly what to fix. The evaluator found issues that unit tests and a rough loop missed, including FastAPI route ordering that could pass tests but break in production and Boolean logic around the delete key. It also tested the application itself, including play mode. The resulting game had a live debug HUD, working arrow-key movement, and collision with castle walls. Ash attributes the difference between the solo run and the harness run to the scaffolding, while admitting that the harness was expensive and took about six hours for the example.
Trace reading is the main way to improve an evaluator
Ash is direct that Claude is a poor general QA agent out of the box. Its generosity bias leads it to find a bug and defer it instead of insisting on a fix. The team improved the evaluator by reading traces, locating where its judgment differed from a human judgment, and changing the prompt. He compares this to reading a stack trace. The team sometimes pipes transcripts into files and uses another agent to search or replay them, but manual reading remains the strongest method. Ash says engineers need to understand why the model made a decision, then change prompts, skills, rubrics, or other harness components based on that behavior.
A stronger model can make parts of the harness unnecessary
Ash says the harness must match the model's current weaknesses. Context resets were important for Opus 4.5 because it showed context anxiety, while Opus 4.6 could maintain a coherent two-hour build with one continuous session and compaction. Earlier versions ran the evaluator after every sprint. Later versions ran it after a complete one-shot generation. The lesson is not that the earlier harness was wrong. It was right for an earlier model. The final pattern kept the planner, generator, and evaluator while removing other components and using the file system for shared state. Ash says the simplified version was roughly half the cost of earlier runs, though still expensive.
