Production agent failures often cannot be reproduced by rerunning the same prompt because model serving and execution conditions vary.
2
Recording inputs and outputs at workflow boundaries lets engineers replay a failed run without making new model calls.
3
A recorded trace can become a deterministic test case, with selected nodes stubbed and changed tools or guardrails run live.
Summary
Tisha Chawla and Susheem Koul argue that production agents need replayability rather than bitwise deterministic model output. Setting temperature to zero does not remove variation from floating-point operations, request batching, or mixture-of-experts routing, and it also cannot correct a bad reasoning path. Their Chronicle proof of concept records the inputs, outputs, metadata, and state around annotated workflow boundaries, including LLM calls, tools, and retrieval. A failed stock-order trace shows exactly how an agent turned a dollar amount into a share quantity. Engineers can then replay that trace, stub unchanged nodes, run a modified tool or guardrail live, and assert that the failure is blocked. The talk separates deterministic testing of tools and guardrails from behavioral testing of agent tone and trajectories. The practical advice is to record the full execution envelope, preserve variables such as model version and build ID, and use production traces as regression tests.
Rerunning the same prompt usually loses the production failure
The speakers open with an agent that called the wrong tool and wrote the wrong data. The usual response is to pull the prompt from telemetry, send it to the same model, and run it locally. That rerun may work repeatedly, while the costly production failure disappears. Their stock example makes the risk concrete: a user asks the agent to sell $1,000 of stock, but the agent sends 1,000 as the quantity. At $190 per share, the order becomes a $190,000 mistake. The broker API returns HTTP 200 in 30 milliseconds, with no exception or alert, so ordinary dashboards remain green.
Temperature zero does not make the whole agent deterministic
The speakers reject temperature zero as a solution. Greedy decoding can repeat the same bad reasoning path, and hosted model APIs still have lower-level sources of variation. They cite floating-point addition order, request batching, and mixture-of-experts capacity limits. A request can be grouped with different traffic, which can change routing and final token scores. Their point is that identical text output is not a practical production target, especially when the model is hosted and changing. The system needs a way to inspect and rerun the execution that already happened.
Replayability records a run for debugging instead of freezing the model
The talk separates bitwise determinism from replayability. Bitwise determinism means the same input produces the same output, but the speakers call that controllability and say engineers do not get it from a hosted API. Replayability means re-validating a run that already happened well enough to debug it. The model can remain variable. The important artifact is the recorded run, including the decisions and state transitions around it. This changes the question from how to force model determinism to how to debug and retest an execution that cannot be reproduced naturally.
Recording belongs at workflow boundaries rather than the network layer
Network logs miss local retrieval, in-process tools, memory operations, streaming behavior, and asynchronous work. The proposed recording point is the boundary around each workflow node, where the system can capture what enters and leaves the method. A node may be an LLM call, a tool call, or a RAG retrieval. The resulting trace records the meaning of each step instead of only the packets exchanged. The speakers describe a workflow that moves from annotation and recording through visualization and diagnosis to fixing, replay, and verification.
Chronicle freezes annotated nodes into an execution trace
Chronicle is the speakers' proof of concept for recording agent workflows. A boundary annotation wraps a method with a bounding box and records its input and output. The annotation can also capture session details such as model version and code version, so the state of the run is saved as a trace. In the stock example, planning, the place-order tool, and the final response method are annotated. The trace shows that the LLM produced a call with symbol Acme and quantity 1,000, then shows the tool executing that input.
A failed trace can test a fix while leaving the rest of the run unchanged
Once the trace identifies the bad tool call, engineers can add a guardrail to the tool and test it against the recorded execution. Replay mode stubs the nodes that should remain unchanged and runs the changed node live. In the example, the planning LLM is stubbed with its recorded input and output, while the tool runs with the new guardrail. The tool blocks the order, and an assertion checks that result. This preserves the original stack trace while testing the code change that matters, without asking the model to make a new decision.
Deterministic and behavioral tests cover different parts of an agent
The speakers divide agent testing into deterministic and behavioral testing. Deterministic testing fits tools and guardrails, where Chronicle can stub LLM outputs and replay the complete run without model calls. These tests are rerunnable and free of model-call costs. Behavioral testing measures subjective properties such as the agent's tone or whether its trajectory was appropriate. For those questions, they recommend methods such as an LLM judge. A replay trace therefore supports regression tests for concrete state changes, while behavioral evaluation remains a separate activity.
The full execution envelope matters more than the prompt alone
The closing advice is to stop chasing bitwise determinism through an API and log the variables that shaped each session. The speakers name LLM version, build ID, and retrieved RAG chunks. They also say to capture the full envelope around a response rather than saving only the prompt. After finding and fixing a failure, engineers should replay it and keep the same trace as a test case. They advise keeping generation-time variation instead of pinning temperature to zero, since that variation is part of the agent's behavior.
"Record at the boundary instead because you need to capture what enters each node and what leaves it."06:20
Who should watch
You operate agents against production systems where an incorrect tool call can mutate enterprise or customer data.
Your team can inspect logs but cannot recreate the exact model, retrieval, and tool path behind an incident.
You want to turn real agent failures into regression tests for guardrails and tool changes.
