# Taming Rogue AI Agents with Observability-Driven Evaluation Jim Bennett, Galileo | AI Engineer World's Fair 2025 | 16:15 Source: https://www.youtube.com/watch?v=xJXm4Wcw4m8 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/taming-rogue-ai-agents-with-observability-driven-evaluation Published: 2025-06-27 Tags: evals, human-in-the-loop, observability, tracing ## TL;DR - AI agents are hard to test with ordinary unit tests because their outputs vary and their workflows span multiple model, retrieval, agent, and tool calls. - Evaluation should measure each step of an agent workflow, including tool use, retrieval, hallucination, task completion, and progress toward the final goal. - Human feedback is needed to check and improve AI-generated metrics, while evaluations should run during development, in CI/CD, and in production with alerts. ## Summary Jim Bennett argues that AI applications need evaluation and observability from the start because ordinary unit tests cannot capture the behavior of nondeterministic, multi-step agent workflows. He uses a banking chatbot to show the difference between completing a task and making progress toward it: the chatbot eventually finds an account balance, but only after several turns. Bennett recommends tracing every model call, retrieval step, agent decision, and tool call, then applying metrics at each level. A stronger model, or a model trained specifically for evaluation, can score traces and suggest likely fixes. Those scores still need human review because the evaluator can be wrong. Bennett's workflow puts evaluations into prompt engineering, model selection, CI/CD, and production monitoring. Production alerts can notify a team when an agent starts failing or behaving unexpectedly. ## Key ideas ### AI agents cannot be tested like deterministic functions [02:36](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=156s) Bennett contrasts a simple unit test for an add function with an agentic workflow whose output depends on multiple calls and decisions. A request can pass through an LLM, retrieval, another agent, a tool, and more model calls. Even defining success is difficult for a conversational system. The question is not only whether the final answer looks acceptable. Engineers also need to understand what happened across the workflow and where the system stopped making progress. ### One AI system can evaluate another AI system [03:35](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=215s) Bennett applies the idea of "set a thief to catch a thief" to AI evaluation. An LLM can inspect an application's input, retrieved information, and output, then score whether the result worked. He says AI evaluators perform about as well as humans at judging whether an AI system worked. The evaluator should generally be a better model than the one running the application. The application can use a cheaper model, while evaluation uses a stronger or specially trained model. ### Evaluation needs metrics at every step of the workflow [06:41](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=401s) A useful evaluation collects the data moving through the application and defines what to measure at each stage. Bennett names tool-call success, retrieval quality in a RAG system, sensible answers, and hallucinations as examples. In a multi-agent banking application, the orchestration agent, another agent, and a tool can each fail independently. Looking only at a final yes-or-no result hides that distinction. Traces and per-component metrics show where the failure occurred and which team should address it. ### Action completion and action advancement measure different outcomes [09:49](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=589s) Bennett uses two metrics for the account-balance chatbot. Action completion asks whether the system completed the requested task from input to output. Action advancement asks whether it moved toward the final goal. The first request, "What is my account balance?", neither completes nor advances because the bot refuses the request. Asking for the checking-account balance advances the conversation because the bot asks for the account name, even though it has not returned a balance. The final turn completes the task. ### Trace-level observability connects failures to fixes [11:06](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=666s) A trace can show an LLM call, the model's decision to call a tool, the data returned by that tool, and the next LLM step. Bennett says engineers should see an overall failing result and then open the individual steps to understand why it failed. This level of detail can point to different owners and remedies. A retrieval system may need work, or a prompt may need tuning. The architecture of the agent must be visible enough to support that diagnosis. ### AI-generated analysis can suggest changes, but should not make them alone [12:18](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=738s) Bennett describes an analysis that found the chatbot sometimes failed to use the get balance tool when users asked about account balances. It suggested adding explicit instructions to the system message, such as directing the agent to retrieve balances for all relevant accounts. He does not recommend allowing the system to apply fixes automatically. An automatic change could create a new failure, so a human should review the suggestion and then run the evaluations again. ### Human feedback is needed to train and check evaluation metrics [13:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=828s) AI-generated metrics are not guaranteed to be correct. Bennett recommends a system with human feedback, which he calls continuous learning by human feedback. People review scores, decide whether the metric reflects actual behavior, explain why a score is wrong, and use that feedback to retune the evaluator. Metrics will not be perfect out of the box, so their quality also needs ongoing review and training. ### Evaluations belong in development and production [14:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=865s) Bennett's process starts by adding evaluations before, or immediately after, building an agent. Teams should define measurements during prompt and application design, then carry them through prompt engineering, model selection, CI/CD, and production. The measurements depend on the application and may cover toxicity, hallucinations, comprehensibility, retrieval, or a custom use case. Since users produce inputs developers did not anticipate, production monitoring and real-time alerts are needed when an agent goes rogue. ## Notable quotes - "We can actually ask a non-determining system like an AI to evaluate an AI for us." (03:55) - "It's not just that binary did my agent work yes or no question. It's at what step in the process did my agent fail." (07:31) - "The best time to put evaluations in is as you're doing prompt engineering model selection. The second best time is now." (09:19) - "You want humans to evaluate the numbers and say, "Okay, this is actually working."" (14:05) ## Tools & references mentioned - Galileo - Chicago Sun-Times - Atonement - Ian McEwan - The Martian - Andy Weir - Butler Snow - Air Canada - RAG - continuous learning by human feedback ## Who should watch - You are building a multi-step agent that calls models, retrieval systems, other agents, or tools and need to find which component is failing. - Your application is moving from prompt experiments toward CI/CD or production, and you need metrics that can track behavior beyond a final pass or fail. - You need a human review process for AI-generated evaluations and alerts for unexpected production behavior. ## Related talks - [Fighting AI with AI](https://aietalks.com/talks/fighting-ai-with-ai) (Lawrence Jones, Incident.io, 17:29) - [Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents](https://aietalks.com/talks/break-it-til-you-make-it-building-the-self-improving-stack-for-ai-agents) (Aparna Dhinakaran, Arize, 14:25) - [Mind the Gap (In Your Agent Observability)](https://aietalks.com/talks/mind-the-gap-in-your-agent-observability) (Amy Boyd & Nitya Narasimhan, Microsoft, 1:20:07) - [Production Evals For Agentic AI Systems](https://aietalks.com/talks/production-evals-for-agentic-ai-systems) (Nishant Gupta, Meta Superintelligence Labs, 08:12) - [Ensure AI Agents Work: Evaluation Frameworks for Scaling Success](https://aietalks.com/talks/ensure-ai-agents-work-evaluation-frameworks-for-scaling-success) (Aparna Dhinkaran, Arize AI, 15:28)