Agentic systems need evaluation of planning, tool use, execution, recovery, and complete workflows, rather than model answers alone.
2
Production traffic provides the most representative evaluation data through traces, outcomes, escalations, failures, and user feedback.
3
Evaluation should run continuously inside the production control plane, with automated checks, telemetry, simulations, and targeted human review.
Summary
Nishant Gupta argues that benchmark scores do not tell teams whether an autonomous system will behave reliably in production. Agents plan, call tools, retrieve information, execute workflows, and recover from failures, so evaluation must cover the whole workflow. Offline tests still matter, but they should use realistic scenarios such as customer support, code generation, and research workflows. Once an agent is deployed, its production interactions become evaluation data. Traces, user outcomes, escalations, failures, and feedback show how the system behaves under real conditions. Gupta also describes a continuous loop in which human reviewers assess edge cases, feedback improves datasets, and offline scenarios validate changes. The proposed architecture places evaluation in a control plane that observes the execution plane, collects telemetry, runs simulations, and coordinates review. Metrics such as task completion, tool success, escalation, safety violations, latency, cost, and recovery rate connect system behavior to business outcomes.
Gupta says traditional evaluation asks whether a model generated the right answer. Agentic systems require a broader question: whether the system behaved correctly. An agent may plan, call tools, retrieve information, execute a workflow, and interact with production infrastructure. Evaluation therefore needs to cover planning quality, tool usage, execution, recovery from failures, and decision-making. A high answer score can hide failures in the surrounding workflow. The unit being evaluated shifts from an individual output to the complete operation the system performs.
Production has failure modes that benchmarks do not capture
Offline benchmarks can improve while production reliability remains unpredictable. Gupta attributes the gap to conditions that static tests do not represent, including tool failures, API outages, changing context, user variability, and long-running workflows. He describes a hierarchy of failures that begins with memory, retrieval, and safety problems, then includes reasoning mistakes, poor planning, and incorrect tool execution. At the highest layer are failures in coordination between multiple agents. Evaluating only the final model output misses much of this operational risk.
Reliability is a better operating target than benchmark accuracy
Gupta asks teams to think like site reliability engineers and production engineers. They should measure reliability, availability, latency, cost, and recovery instead of treating accuracy as the main measure of success. He presents a pyramid with benchmarks at the bottom, scenario-based evaluations in the middle, and production telemetry at the top. Benchmarks remain useful because they are scalable and repeatable, but the strongest evaluation signals often come from real users interacting with real systems. The goal is dependable outcomes rather than the highest benchmark score.
Offline tests should model scenarios instead of isolated prompts
Offline evaluation still has a place, but Gupta says its method should change. Teams should evaluate scenarios such as customer support, code generation, and research workflows. The agent operates inside a simulated environment, where the team measures task completion rate, tool correctness, planning quality, and resource usage. This approach tests how parts of the system work together. It also gives teams a way to validate changes before or alongside production monitoring. The central shift is from testing prompts to testing realistic tasks and environments.
Production interactions become a continuous source of evaluation data
After deployment, every interaction can provide a signal about system behavior. Gupta lists execution traces, user outcomes, escalations, failures, and feedback as useful data. Production traffic is more representative than a fixed test set because it contains the variety of situations users actually create. This data can reveal problems that offline scenarios missed. It also gives teams evidence about whether the system completed the task, created extra work for people, or required escalation. Production is therefore part of the evaluation system rather than only the place where the agent runs.
Human reviewers provide signals automated metrics miss
Gupta rejects the idea that humans are merely fallback systems. He calls them evaluators who can assess correctness, trust, usefulness, and safety. Human feedback helps calibrate automated evaluation pipelines and find blind spots in their metrics. He recommends combining automated evaluation with targeted human review rather than sending every interaction to people. Review can focus on edge cases and uncertain behavior, while automated systems handle broader monitoring. This gives teams qualitative judgments where simple success or failure signals are insufficient.
Drift requires monitoring after every system change
Agent systems can change when models, prompts, tools, or user behavior change. Gupta says a single change may not look catastrophic even when reliability is gradually worsening. Teams may see success rates decline, escalations increase, and tool failures rise over time. Without continuous evaluation, they may not notice the drift until users complain. Monitoring therefore needs to continue after deployment. It should compare behavior over time and expose changes in outcomes, tool performance, and the amount of human help required.
Agent traces make evaluation possible at workflow level
Evaluation needs visibility into an agent's reasoning paths, tool calls, memory access, execution timelines, and state transitions. Gupta says traditional logs are insufficient for this level of analysis. Detailed agent traces provide a form of distributed tracing for autonomous workloads, much like tracing is used across deeply nested microservice architectures. Without this observability, teams are forced to guess why an agent succeeded or failed. The traces connect an end result to the sequence of actions that produced it.
Evaluation belongs in the production control plane
Gupta describes evaluation as an always-running service rather than a testing phase before deployment. Human reviewers examine edge cases, feedback improves datasets, offline scenarios validate updates, and the loop continues after release. In the architecture he presents, a control plane observes the execution systems, collects telemetry, runs simulations, and coordinates human review. The execution plane performs the work. This separation lets evaluation measure and govern system behavior continuously instead of operating as a separate offline QA tool.
"The question is no longer did the model generate the right answer? The question is did the system behave correctly?"00:42
Who should watch
You are building an agent that calls tools or runs multi-step workflows, and your current tests mostly check final answers.
Your team sees benchmark scores improve while production failures, escalations, or tool errors remain hard to explain.
You need an evaluation design that combines offline scenarios, production telemetry, automated checks, and targeted human review.
