Evals must change as agents gain tool calls, reasoning, memory, dynamic interfaces, and longer task loops.
2
Deterministic checks and LLM judges still have a place, but they cannot catch every failure in agents with changing trajectories.
3
Agent as a judge can inspect traces for unexpected patterns, then create a pull request with a possible fix.
Summary
Aparna Dhinakaran argues that evals have to evolve with the systems they measure. Early agents mainly answered prompts, while newer agents use tools, reason over longer tasks, start sub-agents, and produce different trajectories for different users. Those changes create failures that fixed checks and fixed LLM judging rubrics can miss. She describes three layers of evaluation: deterministic checks for things that can be defined in advance, LLM as a judge for analysis against a stated rubric, and agent as a judge for adaptive analysis across traces. Arize's Signal is a long-running agent that searches traces for patterns such as repeated tool calls, inefficient trajectories, and loops. It can also open a pull request with a fix. Dhinakaran's position is that teams should use all three forms of evaluation, because each handles a different class of problem.
Evals have become part of the normal work of serious AI teams
Aparna Dhinakaran says evals moved from being a new skill for product managers and AI engineers to something every serious AI team is betting on. Arize works with teams before agents ship and also evaluates live production traces. The company runs over 100 million evals each month. The average team runs about 12 eval jobs, while its top teams run more than 3,800 evaluators. Dhinakaran focuses on trace-based evaluation because it helps teams see what works, find failures, and supply data for continual learning loops.
Agent complexity changed the failures that evals have to find
Dhinakaran describes a series of changes in the systems being evaluated. In 2023, an agent mainly answered a prompt. In 2024, frontier models added tool calls, reasoning, and deep research. Teams now run loops on real-world data, with sub-agents handling long-horizon tasks. Each step added complexity and produced a different kind of problem. The evaluation target was changing while teams were still using first-generation evals, so the old checks could no longer describe every way the system might fail.
A production agent exposed failures that fixed judging missed
Arize built an agent called Alex into its own product, giving the team direct experience with these problems. As new capabilities appeared, Alex gained longer memory, dynamic user interfaces, and the ability to search a large volume of traces. It also forgot context, failed to know when work was finished, and sometimes became stuck in loops. Dhinakaran says classical LLM-as-a-judge evals could not catch all of these failures. Each user interaction could produce a new interface and a different trajectory, which made a fixed evaluation flow a poor fit.
The right judge depends on the kind of failure being tested
Dhinakaran's proposal is to evaluate an agent with another agent when the behavior is adaptive and difficult to describe in advance. This does not remove deterministic evals or LLM-as-a-judge evals. Deterministic checks still test conditions that can be specified directly. An LLM judge applies a fixed rubric and fixed scores. An agent judge performs adaptive, dynamic analysis, which fits systems that take different paths each time a user supplies data. She expects the future to include all three methods rather than one replacing the others.
Signal searches traces for patterns that teams did not write checks for
Arize released Signal as an agent-as-a-judge system. Dhinakaran describes it as a long-running agent that reads submitted traces and discovers patterns in their problems. It can identify subtle issues that a classical LLM judge using deterministic rubrics would not find. Her examples include an operation repeating inside a loop, the same tool being called repeatedly for too long, and an inefficient trajectory. These are behaviors that may only become visible across the full trace rather than in one fixed score.
Agent judges can connect evaluation to a code change
Signal does more than report an issue. After analyzing traces and finding a problem, it can open a pull request with a fix. Dhinakaran presents this as part of the new evaluation workflow: the judge searches for failures that were not anticipated, identifies the pattern, and proposes a change in the code. The talk does not claim that every proposed fix will be correct. It describes the ability to move from trace analysis to a concrete pull request.
"Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on."00:33
Who should watch
You are building or operating an agent whose behavior changes across users, tools, or long-running tasks, and fixed test cases are missing failures.
Your team already uses deterministic checks or LLM judges and wants to understand where an agent-based evaluator fits.
You work on production observability and want evaluation findings to lead to proposed code changes.
