AI agent development becomes a data flywheel after launch, where traces, user feedback, and failures produce targeted evals and product fixes.
2
Unit, trajectory, and rubric-based evals serve different purposes, but fine-grained unit evals can overfit to one model's way of completing a task.
3
User satisfaction remains the final test, so offline scores should lead to controlled A/B tests that measure feedback, activation, and retention.
Summary
Rafal Willinski and Vitor Balocco describe how Zapier builds an evaluation system for agents that behave unpredictably and encounter user behavior the team cannot anticipate. They begin with detailed tracing of model calls, tool calls, errors, and processing steps so individual failures can become repeatable eval cases. Explicit feedback helps, but interaction signals such as enabling an agent, copying a response, repeated prompts, and expressions of frustration provide more data. The talk then compares unit-style evals, trajectory evals, rubric-based LLM judging, and A/B tests. Zapier found that an extensive collection of fine-grained unit evals could make stronger models look worse because the tests rewarded one path through a task. Their current approach uses each eval type for a different purpose. They also separate regression cases from difficult aspirational cases and treat live user behavior as the final measure of quality.
The first prototype is only the beginning because both the model and its users behave unpredictably. After launch, the team needs to collect feedback, understand usage patterns, identify failures, and turn those findings into evals and product changes. Better fixes attract more users, which creates more usage and more failures to study. Willinski describes this as a cycle that keeps producing work. The point is to make failures useful rather than treating them as isolated support problems. An agent platform for nontechnical users is especially difficult because the team cannot predict all the ways people will describe tasks or combine tools.
Tracing only completion calls is not enough. Zapier also records tool calls, tool errors, preprocessing, and postprocessing so engineers can reconstruct the full run. The trace should use the same data shape as runtime data, which makes it easier to turn a real run into an eval by prepopulating inputs and expected outputs. This matters when tools have side effects. During evals, those effects can be mocked from the recorded trace instead of sending an email or changing a real system. The trace therefore supports both debugging and the creation of repeatable test cases.
Implicit user behavior supplies feedback when buttons do not
Explicit thumbs-up and thumbs-down feedback is useful, but users rarely provide it, especially in detailed form. Zapier looks for signals in the product experience. Turning an agent on after testing it is treated as positive feedback, as is copying a model response. Repeatedly restating a request can indicate that the first answer failed. Messages such as telling the agent to stop 'slinging around' provide negative feedback, as can user frustration and cursing. Zapier also uses an LLM to group frustration into a weekly Slack report, although the team had to tune it to understand frustration in the context of its products.
Internal tooling should turn interesting runs into evals quickly
The speakers recommend using or building LLM operations software that can connect the parts of an agent run, including model calls, database interactions, REST calls, and tool calls. They build both external and internal tooling. Internal tools can reflect Zapier's own domain and should make it possible to turn an interaction or failure into an eval with almost no friction. Once engineers can inspect individual runs, they can aggregate feedback, cluster interactions, and group failure modes. These groups show which tools and interaction types cause the most trouble, creating a practical roadmap for where to spend engineering time.
Unit-style evals predict the next state from the current state. They can check whether the next action is the right tool call, whether parameters are correct, whether an answer contains a required keyword, or whether the agent knows it is finished. They are easy to add and help teams develop the habit of finding a real failure, reproducing it, and fixing it. The speakers warn against converting every positive feedback event into a test. Unit evals work best when they target specific failure modes found in actual data, rather than trying to describe every successful interaction.
Fine-grained tests can make stronger models look worse
Zapier found that its new models could be objectively stronger while scoring worse on internal benchmarks. The team had built so many fine-grained unit evals that comparisons became noisy and hard to interpret. Different models may reach the same goal through different tool sequences, but a unit eval can penalize every path except the one encoded in the test. The dataset had therefore overfit to the existing models and the data collected with them. Reasoning models helped compare runs and explain differences, such as one model acting decisively while another asked more follow-up questions or produced invalid JSON.
Trajectory and rubric evals cover behavior that unit tests miss
Trajectory evals let an agent run to its end state and grade the tool calls and artifacts created along the way, rather than checking only one iteration. They are more realistic but harder to set up, especially when tools have side effects. Zapier chose to mirror a user's environment with a synthetic copy instead of mocking the whole environment, because mocks can produce unrealistic data. These evals can also take much longer to run. For broader comparisons, Zapier uses LLM judges with human-written rubrics that tell the judge what to inspect for each example, such as whether the agent retried after an unexpected calendar API error.
A high eval score does not prove that an agent is good. If a team approaches a perfect score, the dataset may simply be too easy or too narrow. Zapier separates regression cases, which protect existing customer use cases, from aspirational cases that are deliberately difficult. The final judge is user satisfaction, so the speakers prefer controlled A/B tests for verification. A small share of traffic can receive a new model or prompt while the team monitors feedback, activation, retention, and related metrics. This connects offline evaluation to what users actually experience.
"Building good AI agents is hard and building good platform to enable nontechnical people to build AI agents is even harder."00:35
Who should watch
You are shipping an AI agent and need a repeatable way to turn production failures into tests.
Your offline eval score is improving, but you do not know whether users are happier or whether the test set has become too easy.
You are choosing between unit tests, trajectory tests, LLM judges, and live experiments for a tool-using system with side effects.
