Teach Your Agent to Prove It's Done

Nick Nisi, WorkOS, Zack Proser, WorkOS1:01:04 · Oct 2026 · 13K views
Thumbnail for Teach Your Agent to Prove It's Done Watch on YouTube
TL;DR
  1. 1

    Agents need explicit goals and tools that let them observe whether those goals were reached.

  2. 2

    A deterministic verifier CLI can run health checks, browser actions, persistence checks, screenshots, and other evidence-producing tests in a repeatable loop.

  3. 3

    Failures should feed the next repair automatically, while receipts and eval baselines preserve evidence of what the application does over time.

Summary

Nick Nisi and Zack Proser argue that an agent's claim that work is finished is not enough. Developers need to define what "done" means, give the agent tools to inspect the application, and make those checks repeatable. Their Midnight Snack Club demo shows how a CLI can start the app, check its health, drive a browser, inspect storage and console output, take snapshots, run deterministic proofs, and ask an LLM judge for a looser assessment. A failed check should send the agent back to repair the implementation instead of leaving the failure for a human to discover. They also recommend keeping receipts such as screenshots, videos, test results, and run data. Over time, those records can support eval baselines and regression checks. The talk is practical about the limits: a screenshot can lie, an LLM judge can miss a broken contract, and the verifier must fit the application.

Key ideas
03:28

Human review cannot keep pace with agent-written code

Nick Nisi says reviewing every line or pull request becomes less useful as agents write code faster and developers become less familiar with the resulting code. A browser check can confirm that a form submission shows a success message, but the Midnight Snack Club demo exposes the limit. The email appeared on the page, then disappeared after reload because it was never persisted. Zack Proser describes this as the subtle bug that appears when someone is managing many terminal sessions and agents. The code can build, run, and pass tests while still being wrong from a user's point of view.

09:28

An agent needs a defined outcome and a way to inspect it

Nisi says every unspecified detail about the desired outcome becomes an assumption the agent may invent. The remedy is to define a clear goal and give the agent a way to observe and check that goal. Their preferred evidence includes screenshots, videos, test results, and smoke tests. The result matters more than whether the code looks like something the developer would have written. Proser describes the desired pattern as assigning each task an agent with enough information to know when it is done and how to verify that state. The agent can choose its implementation while the verifier checks the result.

11:38

A centralized verifier can turn evidence into workflow state

Proser describes CTX, a web service running on Cloudflare that began as shared memory for different agents and became an experiment in centralized verification. An agent reports completed work through an MCP connection and asks CTX to verify it. A verifier can call the GitHub API to check that a pull request exists, is healthy, and has passed its merge checks. Only then does the workflow mark that node satisfied. If checks fail, the agent receives the remaining failures and is redirected to repair them. Proser also describes Ryker, a tool that watches work, runs the required checks, retains evidence, and alerts him only when human attention is needed.

18:28

Application-specific CLIs make feedback repeatable

The presenters recommend turning manual developer checks into scripts instead of relying on a vague instruction such as "always verify the change in a browser." A script can perform deterministic checks, return precise failures, and let the agent run the repair loop repeatedly. Their workshop tool, Snack, can start and stop the app, run a doctor check, drive the waitlist, reload the page, reset data, inspect storage and network state, read console logs, take snapshots and screenshots, and run proofs or judges. The tool is a template rather than a universal product because each application has different behavior and different things worth checking.

28:28

Feedback-loopable software exposes state the agent cannot see

Proser uses a bouncing-ball canvas as a deliberately simple example. An agent cannot see the running canvas, but the application can print the ball's coordinates at each point. If the ball suddenly moves more than one pixel per loop, that output gives the agent a detectable symptom. This makes a visual behavior available to an automated feedback loop. The presenters say the same pattern applies to application-specific performance, database operations, networking, rendering, and user flows. Once the behavior is measurable, agents can compare variants against the same eval and select an implementation without requiring a person to inspect every version.

20:50

Deterministic proofs and LLM judges answer different questions

Snack has a proof command for checks with exact expected results and a judge command that uses a system-one model when a strict text match would be too brittle. Their waitlist proof checks that the page starts empty, enters an email, clicks the join button, sees the expected status, and confirms that the email appears. The judge can assess whether a result is going in the right direction even when wording differs. The presenters demonstrate the danger: the app may show a confirmation that sounds correct while violating the expected contract, and the judge may miss the issue. They advise using both deterministic checks and looser evaluation where appropriate.

32:02

A failure should start the next repair without waiting for a human

Once a loop is durable and repeatable, a failed run can drive future work. The agent can identify that the persistence fix is incomplete, make another change, rerun the command, and stop only after the evidence passes. Nisi shows this moment in the demo: the app accepts an email, but storage is empty after reload, so the agent knows it is not done. Detailed error messages help because an agent can interpret the failure and act on the state that caused it. The goal is for developers who are managing many concurrent work lanes to return to completed work instead of restarting each task from the last failed check.

37:59

Receipts turn past runs into eval data

The presenters recommend keeping receipts from agent runs, including user flows, screenshots, recordings, timing, errors, and other state. Those records can help fix current problems and later become data for comprehensive evals. A team can create an eval baseline, run it consistently, and update it as new failures appear. The baseline can show whether a change is safe, improves the application, makes it worse, or changes nothing. Previous checks also become regression coverage for new features. The speakers describe this as building an evidence ladder, where unit tests, smoke tests, evals, and application-specific verifiers accumulate into a living specification.

"This is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done and I'm not going to drop this work on the floor."Zack Proser36:51
Who should watch
  • You run several coding agents or terminal sessions and need them to finish work without waiting for constant human nudges.
  • Your web application has tests, but agents still report success when persistence, rendering, or user behavior is wrong.
  • You want to build an application-specific verifier, eval baseline, or receipt system that can support faster local development and safer pull requests.