Agent traces record telemetry, but they do not preserve the runtime state needed to replay an execution.
2
Checkpoints let teams fork production runs, change a model or tool, and compare the result against the original.
3
Replay should run across cohorts of real cases because a single cheaper model result can hide worse task performance.
Summary
Hamza Tahir argues that agent infrastructure needs a durable checkpoint connected to the runtime, rather than relying on read-only traces. A checkpoint captures state around the code, files, artifacts, environment, and tool calls, so an execution can resume after a failure or be replayed from a chosen point. Teams can then swap a model, mock a tool, or change a policy and compare the fork with the original run. Tahir demonstrates this with Kitaru, replaying a support agent and diffing the resulting executions. He also describes a cohort workflow based on real production runs, with an MCP server helping analyze large JSON reports. The talk is honest about the risk of naive optimization. A cheaper model can reduce token cost while making support outcomes worse. The recommended loop is to checkpoint production runs, select meaningful cohorts, replay changes, compare results, and keep a human involved before shipping.
Tahir distinguishes an observability trace from a durable save point. A trace records emitted telemetry about tool calls and inputs and outputs, but it does not preserve the variables in state, the in-flight file system, code decisions, or the code itself. Those details are left as a read-only record in a separate tool. A useful save button must connect observability data to the runtime that executed the agent, so the system can recover the state rather than merely inspect what happened.
With runtime checkpoints, a team can ask what would have happened after changing one part of an execution. Tahir gives several examples: swap in a cheaper open-source model, mock a tool with a different return value, or intentionally degrade part of the system to test its behavior. For a customer-resolution agent handling chargeback disputes, checkpoints could help test order-status changes, escalation decisions, or whether a smaller model would have been sufficient.
Replay closes the loop from production behavior to evaluation
Tahir describes a loop of checkpoint, replay, diff, decide, and ship or route. Teams can select a cohort of runs because they were expensive, slow, or risky, replay a change against those real executions, compare the new results with the baseline, and use the evidence in an evaluation process. He presents this as a way to evaluate with production checkpoints instead of relying only on synthetic examples.
DoorDash used replay to speed up customer-bot simulations
Tahir cites a DoorDash blog post about a simulated environment for replaying customer bots and running what-if scenarios. He says the process went from taking hours to taking five minutes with hundreds of simulations, produced 90% fewer hallucinations, and remained within two points of what DoorDash had seen in production. His point is that simulations grounded in previous executions can be both faster and closer to real behavior.
Kitaru snapshots the execution around each checkpoint
In the demonstration, Kitaru provides a runtime layer below the agent harness and connects it to traces. The interface shows a timeline of tool calls and checkpoints. At a checkpoint, the user can inspect the configuration, execution location, code, and artifacts. The snapshot also includes the surrounding environment, such as a Docker image or sandbox. This connects the visible trace to the material needed to continue or fork the run.
Tahir replays an execution after a tool call while changing the model to GPT-5 Nano. Kitaru launches a new execution from the selected point and skips the first three checkpoints because their state is already available. The fork only recomputes the changed part and everything after it. He then repeats the process by mocking a lookup-policy tool with another function from the codebase, keeping the model unchanged.
Kitaru can compare the original execution with multiple forks side by side. The shared prefix remains identical, while later tool calls, timing, artifacts, and final decisions can differ. In Tahir's example, all runs restrict the charge, but two require review while the run with the changed policy is considered safe to answer. The cheaper model runs use fewer tokens, yet the changed policy affects the decision, so cost alone is not enough to judge the replay.
Tahir warns that replacing a model with a cheaper one often looks attractive if the only measure is cost. A support agent might spend less while resolving fewer requests. He cites a BrainTrust study about this false economy and says TowelBench shows that a model passing 60% of the time is self-consistent only about a quarter of the time. One replay is therefore an anecdote. Cohort analysis gives a better view of how a change behaves across many cases.
The production workflow needs cohorts and human review
The proposed playbook starts with real production runs and their saved state. Teams build cohorts that matter, such as expensive, long, or risky cases, then replay changes across the cohort rather than shipping after one or two examples. Results can be shipped, routed, or held, with automation doing much of the work and a human involved at the end. In the demonstration, the cohort verdict was not to ship the cheaper model, even though one replay had looked acceptable.
"A trace gives you the emitted telemetry data of how an agent calls tools in the input and output of that state."00:25
Who should watch
You run agents in production and need to understand why a past execution made a particular decision.
Your team wants to test cheaper models or altered tools against real cases instead of relying on one-off demos.
You are building agent runtimes, sandboxes, or evaluation workflows and need execution state that can be resumed and replayed.
