# Will Agent Evaluation via MCP Stabilize Agent Networks? Ari Heljakka, Root Signals | AI Engineer World's Fair 2025 | 14:11 Source: https://www.youtube.com/watch?v=RVN9HWKmkNU Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/will-agent-evaluation-via-mcp-stabilize-agent-networks Published: 2025-06-03 Tags: evals, mcp, reinforcement-learning ## TL;DR - Agent networks become difficult to stabilize because their behavior is hard to observe, their environments change, and progress toward a goal is difficult to measure after failed attempts. - An MCP-connected evaluation engine can score an agent's output, explain what went wrong, and give the agent feedback it can use to improve. - Teams should first test the evaluation workflow manually, then attach it to agents once they understand the evaluators and can maintain them over time. ## Summary Ari Heljakka argues that complex agent networks need evaluation systems that are continuously used and improved, rather than a static collection of tests added to an agent stack. An evaluation engine can examine factual grounding, goal inference, progress, tool selection, policy adherence, and output quality. Through MCP, an agent can access those evaluators, receive scores and explanations, revise its work, and run the evaluation again. Heljakka demonstrates this first with a marketing message, where Cursor uses an MCP server to find a suitable evaluator, improve the text, and compare scores. He then shows a hotel reservation agent that recommends a prohibited neighboring hotel when MCP is disabled. With MCP enabled, the agent discovers and applies a hotel booking policy evaluator and removes the recommendation. He recommends starting offline and manually before connecting evaluators to agents. ## Key ideas ### Agent swarms become unstable as tasks and environments grow more complex [00:31](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=31s) Heljakka says stable agent swarms could, in principle, solve many knowledge-work problems, but current swarms break down on increasingly complex tasks. The causes include limited observability, changing environments, and the difficulty of testing every situation in advance. It can also be hard to tell whether an agent is making steady progress when it does not complete a task in one attempt. Evaluation is his general answer, although he argues that simply adding an evaluation stack is insufficient. ### Evaluation must cover both what an agent believes and how it acts [02:12](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=132s) The evaluation problem spans several types of behavior. One side checks how the agent represents and discusses reality, including whether its output is grounded in what is true. The other side checks behavior, such as whether the agent inferred the right goals, is progressing toward them, and selects suitable tools. Heljakka says teams can begin with a subset of these checks, as long as they run them consistently rather than trying to solve every evaluation problem at once. ### Evaluation stacks need their own maintenance and improvement process [03:36](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=216s) For a hotel reservation agent, Heljakka suggests evaluators for adherence to the hotel's reservation policy, accuracy of the response, and appropriate behavior overall. These evaluators need a place where teams can create large collections, run them repeatedly, and maintain them over time. The evaluation library must also be improved as business requirements change. Heljakka says the evaluators themselves need optimization because repeated runs cost money and their quality affects the feedback given to agents. ### MCP creates a feedback loop between agents and evaluation engines [04:23](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=263s) The proposed loop is simple: an agent attempts a task, an evaluation engine scores the result, and feedback returns as a numeric score with an explanation of what went wrong or right. The agent uses that information to improve its performance. Heljakka presents MCP, the Model Context Protocol, as the connection that lets an agent access the evaluation engine and its evaluator collections. ### An MCP-connected agent can improve text by selecting and rerunning evaluators [05:11](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=311s) In the first demonstration, Heljakka uses a marketing sentence and asks Cursor through MCP to list available evaluator collections. He selects a smart marketing message quality collection and asks Cursor to optimize the sentence with it. Cursor finds the collection, scores the original message, proposes an improved version, and runs a final evaluation. The displayed measures include persuasiveness, quality of writing, and engagingness, with scores shown between zero and one. ### Agents can discover relevant evaluators without being given each evaluator explicitly [08:50](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=530s) The hotel reservation example uses a simple agent built with Pydantic AI. The Sire Hotel's policy forbids recommending the nearby Acme Hotel. With MCP disabled, the agent recommends Acme after the user hints at interest in a neighboring hotel. Heljakka then enables MCP and runs the same request. The agent calls the evaluation service, finds the hotel booking policy evaluator, and returns a response that no longer mentions Acme, even though the evaluator was not specified directly. ### Manual offline testing should come before live attachment to agents [12:15](https://www.youtube.com/watch?v=RVN9HWKmkNU&t=735s) Heljakka recommends running the MCP evaluation flow manually and offline first, as in the marketing message example. This gives the team a clearer view of how evaluators behave and where they fail. After that understanding improves, the evaluators can be attached to agents. He says teams should choose a platform that supports varied evaluators, maintains their lifecycle, and lets them optimize both the agents and the evaluation system. ## Notable quotes - "The general answer to the question is going to be evaluations or evals, but not in a way that you would think." (01:19) - "What we want to achieve is a stabilization loop where the agent attempts certain task, the output of the task will get evaluated by the evaluation engine, and then feedback in the form of numeric score, and also explanation of what went wrong comes back to the agent." (04:23) - "So, this is basically the same thing that will happen inside your agent, but now we just called it manually to make it easy for us to see." (08:31) - "So, as you can see now, the agent came back and now it's not talking about the Acme Hotel again." (11:45) ## Tools & references mentioned - Root Signals - MCP - Model Context Protocol - Cursor - Pydantic AI - Tina's MCP server - Sire Hotel - Acme Hotel ## Who should watch - You are building agents or agent workflows that need to follow business policies across repeated tasks. - Your current eval setup produces occasional scores but does not give agents actionable feedback or support evaluator maintenance. - You want to test MCP-based agent improvement manually before connecting evaluation services to production agents. ## Related talks - [Agent Evals: Finally, With The Map](https://aietalks.com/talks/agent-evals-finally-with-the-map) (Ari Heljak, Root Signals, 13:31) - [Skill Issue: How We Used AI to Make Agents Actually Good at Supabase](https://aietalks.com/talks/skill-issue-how-we-used-ai-to-make-agents-actually-good-at-supabase) (Pedro Rodrigues, Supabase, 1:18:41) - [Ship Real Agents: Hands-On Evals for Agentic Applications](https://aietalks.com/talks/ship-real-agents-hands-on-evals-for-agentic-applications) (Laurie Voss, Arize AI, 2:04:18) - [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00) - [Are MCPs Overhyped? A Rant about MCPs](https://aietalks.com/talks/are-mcps-overhyped-a-rant-about-mcps) (Henry Mao, Smithery, 07:29)