Evals should target known agent failure modes instead of trying to test every possible failure.
2
Human reviewers should record why an output passed or failed so that this knowledge can later support automated judges.
3
Production traces should feed offline evals, while tool calls that change external systems require captured system state or carefully controlled mock systems.
Summary
Phil Hetzel describes four maturity stages for evaluating agents. Teams often begin with vibe checks, but he recommends recording human justifications alongside thumbs-up or thumbs-down judgments. Those explanations expose domain-specific failure modes and can later support LLM-based judges or deterministic checks. As teams mature, production and UAT traces should become part of the eval data set. Hetzel describes a flywheel in which production traces reveal problems, offline evals test those cases, and the results guide agent improvements. More complex agents introduce tool calls and require evaluation of full traces rather than only final outputs. Context-gathering tools are easier to handle than tools that create, update, or delete external data. For CRUD tools, evals need to recreate the external system's state at the time of the original run without touching production. Hetzel discusses storing system state in traces and using timestamp queries against systems such as vector databases.
Evals should target known failure modes instead of exhaustive test coverage
Hetzel says evals are different from unit tests because they do not need to cover every possible failure. A subject matter expert can identify the specific ways an agent might fail, and the team can build evals around those cases. Exhaustive coverage is impractical because the possible failures are effectively infinite. Spending all the team's time writing tests leaves no time to ship. Eval results also do not need to be perfect in every case. LLM-based judges may be nondeterministic, but they can still be useful when they show whether changes are moving in the right direction.
Early vibe checks become useful when reviewers explain their judgments
Hetzel accepts vibe checking as a starting point, provided that teams document what they see. A human reviewer can inspect a small set of example inputs and mark each response as good or bad. The reviewer should also explain why the response received that judgment. Those explanations extract domain knowledge from the reviewer's head. Later, the team can use that material to build an automated judge. Hetzel recommends making annotation views specific to the people reviewing the traces, because reviewers understand what a useful agent trace should look like and are more likely to evaluate it appropriately.
Human justifications can be turned into scalable judges and code checks
Once reviewers have documented why outputs fail, the team can derive explicit failure modes from those explanations. Hetzel says teams may use an LLM coding tool to help identify the patterns. They can then scale the human knowledge with an LLM judge instead of relying on a small group of experts. LLM judges should themselves be evaluated, since giving a model the role of judge does not make its decisions automatically trustworthy. Some failures are better handled with deterministic code. For example, an eval can fail when an agent makes too many tool calls or uses too many tokens.
Production traces should drive the next round of offline evals
At this stage, the eval data set should include production or UAT traces. Hetzel says teams should think of evals as rerunning production rather than simply running tests. His flywheel starts by capturing agent traces from production, identifying what went wrong through human or automated analysis, and bringing those examples into an offline experimentation environment. The team reruns those cases through an eval and uses the results to decide how to improve the agent. This lets evals support improvement work as well as risk control.
External tools require evaluating the whole agent trace
When an agent starts using external systems, the final answer is no longer enough to evaluate. Hetzel separates context-gathering tools, which retrieve information for the model, from CRUD tools, which create, read, update, or delete information. External interactions add more possible failure modes, so teams may need to inspect the full trace and eventually evaluate individual tool or MCP calls. That requires capturing every step the agent took and making those steps available for inspection. The evaluation target shifts from one model output to the sequence of actions and tool interactions that produced it.
CRUD evals must recreate external state without changing production data
CRUD tools create two hard problems for offline evaluation. The team has to represent the state of each external system at the moment the original eval input was created. It also has to let the agent interact with those systems without overwriting production data. Hetzel says this problem is not completely solved. One approach is to use mock-level APIs that approximate the production environment. Another is to preserve enough external system state inside the trace so it can be injected into the eval task. These techniques allow the eval to work from a captured situation rather than a current, potentially different system state.
Timestamped system queries can reproduce the state behind an original trace
Hetzel describes timestamp queries as another way to recreate external state. If an input entered the evaluation data set at a known point in time, a system that supports version or timestamp queries can be asked for its state at that point. He gives a vector database as an example. Querying the database at the relevant time can reproduce the information available when the original task ran. This is more complex than evaluating a simple model call, but it avoids assuming that the current external system still matches the state captured by the original trace.
LLM judges are acceptable when teams evaluate the judges against human decisions
In the closing question, Hetzel says some evaluation criteria are subjective, which makes LLM judges useful. He does not recommend accepting their outputs without checking them. Teams should run many evaluations on the judge itself and compare its decisions with what a human would decide in the same situation. Because judge outputs are discrete, a team can create a ground-truth data set for this comparison. His framing is that the eval must itself be evaluated. Deterministic graders remain appropriate for objective conditions, while subjective judgments can use an LLM judge that has been tested against human judgment.
"You should probably have some human whether it's the person who built the agent or even better a subject matter expert that really knows what a quality response would look like."07:17
Who should watch
Your team is starting agent evals and is tempted to build exhaustive test coverage before shipping.
You have human review data but have not turned reviewers' explanations into explicit failure modes or automated graders.
Your agent changes external data and you need to reproduce past system state safely during offline evaluation.
