Agent evaluation advice from the chatbot era breaks when models, harnesses, and tools change.
2
The worst behavior an agent can produce matters more for user trust than its most impressive capability.
3
Every issue needs a start date and an estimate of the share of users it affects.
Summary
Ben Hylak argues that improving production agents means raising their floor, the worst behavior they can produce, rather than chasing benchmark scores or impressive edge-case capabilities. Traditional eval advice assumes a chatbot with a mostly known answer space. That approach becomes fragile when agents use tools, explore their environments, and change behavior across models and harnesses. Hylak recommends keeping evals in the codebase as tests, because the agent's prompt is now part of a larger harness made of code and connections. For production issues, teams need to know when an issue began and how many users it affects. He also gives three practical lessons from Raindrop: trace clusters do not reliably define product issues, code-based classifiers can run across production traces, and agents should investigate anomalies that deterministic checks have already surfaced rather than trying to discover anomalies themselves.
Continual learning is still uncommon in deployed agents
Hylak opens by saying that real-world agents and lab products do not show much continual learning. He avoids presenting a universal framework because teams are facing different problems and there are no settled standards. His preference is to discuss what builders actually see working in production. He asks the audience to think about where online evaluation advice has failed them and which parts of their agent workflows are difficult in practice. The talk therefore focuses on operating agents and improving them over time, rather than presenting a collection of fashionable terms or frameworks.
Agent evaluation advice still assumes a chatbot with known answers
A year earlier, Hylak says, many products were still chatbots, and evaluation often meant checking whether a factual answer matched an expected string. If users mostly asked questions with known answers, teams could build fact-checking datasets around those cases. Agents have since been deployed in areas such as finance, healthcare, and defense. They use tools and move through an environment, so their behavior is less predictable. When an agent encounters a roadblock, it may find a creative solution that is useful or harmful. That flexibility makes old evaluation patterns a poor fit.
Large fixed eval sets become fragile when the harness changes
Hylak criticizes the standard advice to build a thousand-example eval dataset. Teams can write tests around specific tool calls, then find that many of them stop being meaningful after changing the model or harness. He gives the example of switching to the Claude Code CLI, after which 80% of those evals may no longer work. Since models, tools, and harnesses will keep changing, he warns against spending months on a fixed eval set that slows product changes. Teams should pursue safety without creating evaluation theater, especially if they would not actually delay a model upgrade to rewrite the suite.
The floor matters because bad actions break user trust
Hylak separates an agent's ceiling from its floor. The ceiling is the most surprising capability the product can achieve. The floor is the worst behavior it can produce. Examples include recommending a competitor, deleting data, or sending an AI-generated email to a customer because the agent had access to email. He argues that these low-end failures are what damage trust. A product may have impressive capabilities, but users will remember behavior that causes direct harm or makes the system feel unsafe.
Because an agent now includes its prompt, code, tools, and connections, Hylak says the prompt is effectively the whole harness. That changes the shape of offline evaluation. Instead of relying mainly on a managed prompt playground, teams should write evals that look like unit tests or end-to-end tests and run them locally. He mentions Vitest evals from Sentry as a package that adds evaluation syntax to Vitest, and he says OpenAI calls a similar idea macro evals. The name matters less than keeping the tests as code alongside the system they exercise.
An issue needs a start date and a user-impact estimate
For each discovered issue, Hylak says teams need two pieces of information: when it started and what percentage of users it affects. A problem that began yesterday prompts an investigation into recent changes, such as a model update or a downstream dependency change. The number of affected users determines whether the issue deserves attention now. Three affected users and 100,000 affected users describe very different operational situations. Since agents can produce an effectively infinite number of problems, these two facts help teams decide which problems to work on.
The right evaluation approach depends on the number and role of users
Before advising a team, Hylak asks how many users it has. A product with millions of users can use experiments, including tests on a small sample of a free tier. A product with five or ten users should take a different approach, especially when it is an internal enterprise application that provides important information. He also distinguishes between products where users are domain experts and products where users depend heavily on the agent. An engineer can delete an incorrect code suggestion, while an AI doctor creates a different level of responsibility for the user.
Trace clusters are useful for exploration but do not define issues
Hylak says clustering traces can help with one-off analysis, such as turning many logs into broad groups, but clusters do not scale well as issue tracking. Their boundaries drift, they are difficult to compare reliably over time, and teams do not control how the groups are formed. Product-specific definitions also matter. A broad cluster called price issues might combine a wrong price quote with a wrong refund calculation, even though their root causes differ. Clustering therefore does not reliably provide the start date, growth, or product-specific issue definition that teams need.
Code classifiers scale better across production traces
Hylak recommends applying code mode to traces. Teams can write classifiers, run them in a sandbox, and apply them at production scale. This gives them more control over what counts as an issue than unconstrained clustering does. The classifier approach can be adapted to the product's own definitions and can operate over the large volume of traces generated by real users. Raindrop has a feature that makes this easier, but Hylak presents the method as something teams can implement themselves.
"Agents are very, very bad at anomaly detection. So don't ask your agent to find anomalies. Ask it to investigate anomalies you've already found."18:54
Who should watch
You are building an agent with tools and are relying on eval datasets designed for simpler chatbots.
Your team sees many production failures but lacks a way to tell which ones are new or widespread.
You want to move evaluation into code and use agents to investigate concrete production signals.
