# Agentic Excellence: Mastering AI Agent Evals with Azure AI Evaluation SDK

Cedric Vidal, Microsoft | AI Engineer World's Fair 2025 | 20:01

Source: https://www.youtube.com/watch?v=J4vPq2i0QzE
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/agentic-excellence-mastering-ai-agent-evals-with-azure-ai-evaluation-sdk
Published: 2025-06-27
Tags: evals, guardrails, multimodal, testing

## TL;DR
- Agent evaluation should begin at the start of development, then continue from model checks through end-to-end system tests.
- Manual review of selected examples helps developers understand model behavior before they trust aggregate evaluation scores.
- The Azure AI Evaluation SDK scales testing with built-in quality, NLP, safety, multimodal, and custom evaluators.

## Summary
Cedric Vidal explains how to evaluate AI agents before giving them more independence in production. He separates model and platform safety protections from application concerns such as system prompts, grounding, and user experience. Developers should first compare models manually, then inspect selected agent outputs before running larger evaluations. His demos use the VS Code AI Toolkit to build a web research agent, run it against a dataset, review outputs, and export results as JSON Lines. For larger test sets, Azure AI Foundry provides evaluators for relevance, coherence, groundedness, fluency, similarity, classic NLP metrics, and safety. Scores can use application-specific thresholds, since an acceptable result for a game may differ from one intended for children. Vidal also shows evaluation of images and multi-turn conversations, where a violent image receives a score that can pass or fail depending on the configured threshold.

## Key ideas
### Evaluation begins before the agent is ready for production
[02:46](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=166s)
Vidal says evaluation should start at the very beginning of an AI development project, because waiting until an application is already built is late. He describes four layers: the model and its safety system, the system message and grounding, the application design, and the user experience. Azure provides platform-specific protections for Azure models, but application developers still need to design safeguards around prompts, grounding, and how people use the system. His broader point is that a foundation model is only one part of agent safety. The application needs its own checks and mitigations.

### Manual examples reveal behavior that aggregate scores can hide
[03:59](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=239s)
Before running evaluations across a dataset, Vidal recommends manually comparing specific model responses. An average score from an automated batch can leave developers unsure how the system behaved on an individual input. In the VS Code AI Toolkit, he compares responses from GPT-4.1 and GPT-4o to a panna cotta recipe request with salted caramel butter. He prefers the GPT-4.1 response and says it is much faster in his comparison. This stage checks an uncustomized foundation model and gives the developer a direct sense of its answers before agent-level testing begins.

### The VS Code AI Toolkit can build and inspect a web research agent
[07:29](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=449s)
Vidal uses the AI Toolkit extension in VS Code to create an agent that extracts event names, dates, locations, and attendee counts from web pages. A generated example configures a system prompt and a Playwright MCP server for web navigation. He then customizes the agent for a Luma event page and runs it with GPT-4.1. The agent navigates the page and returns structured event information. Vidal's example shows the value of checking the complete system, including its browsing behavior and output format, rather than testing only a model prompt.

### Agent evaluation can combine outputs from multiple web pages
[11:30](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=690s)
When Vidal runs the event agent over several inputs, one page does not contain the attendee count. The agent follows a link from the Reactor event page to the Luma page, finds the missing number there, and combines information from both sources. He manually reviews the returned answers and marks them with a thumbs-up or thumbs-down. The AI Toolkit can export the evaluation data as a JSON Lines file, which can then be fed into a more automated system. It can also generate agent code using a preferred framework, including OpenAI Agents, along with the MCP configuration and runtime code.

### Azure AI Foundry supplies evaluators for larger test sets
[13:37](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=817s)
After manual spot checks and small local batches, Vidal moves to automated evaluation at scale. Azure AI Foundry includes AI-assisted quality evaluators for measures such as groundedness, fluency, coherence, and relevance. It also includes classic NLP metrics such as F1 and BLEU, along with risk and safety evaluators. Developers can run evaluations in the Foundry portal or from Python code. The notebook example connects to an Azure AI Foundry project, passes a dataset and selected evaluators to an evaluate function, and produces per-question results rather than one unexplained aggregate.

### Thresholds should match the application's tolerance for risk
[16:03](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=963s)
The evaluators return scores between one and five, and developers can configure the threshold that determines whether a result passes. Vidal says the right threshold depends on the application. A game may accept more violent content than an application for children, so the same score can be treated differently in those settings. He demonstrates this with a multimodal safety evaluation. An image containing nails or pins protruding from a character's head receives a score of four rather than the maximum score, which causes a failure under the current setting. Raising the threshold would allow that result for a use case that accepts violent content.

### The SDK also evaluates multimodal and multi-turn agent behavior
[16:42](https://www.youtube.com/watch?v=J4vPq2i0QzE&t=1002s)
Vidal notes that the evaluation tools can handle multimodal models that mix text and images, as well as multi-turn conversations. This matters when an agent's behavior depends on more than a single text prompt and response. His safety example sends an image for evaluation and examines the system's description of the graphic content alongside its score. The result is meant to be interpreted against a configured policy, rather than treated as an absolute decision that applies to every product.

## Notable quotes
- "Evaluation starts at the very beginning of your AI development project. The sooner the better." (03:11)
- "Before evaluating at scale you need first to cherry pick and look at specific examples." (04:51)
- "The key takeaway the foundational model is just one part. Real safety comes from layering smart mitigations at the application layer." (03:59)
- "Depending on your application, you might want your AI agent to be more or less strict." (16:24)

## Tools & references mentioned
- Azure AI Evaluation SDK
- Azure AI Foundry
- Microsoft
- VS Code
- AI Toolkit
- Playwright MCP server
- OpenAI Agents
- GPT-4.1
- GPT-4o
- GitHub Models
- BLEU
- F1
- JSON Lines
- Azure AI Foundry Discord

## Who should watch
- You are building an agent that browses websites, uses tools, or makes decisions and need a repeatable test process before deployment.
- Your current evaluation method is a few hand-written prompts and you need to move from spot checks to dataset-based testing.
- You need to set different quality or safety thresholds for applications with different levels of risk, including multimodal or multi-turn systems.

## Related talks

- [Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work](https://aietalks.com/talks/engineering-better-evals-scalable-llm-evaluation-pipelines-that-work) (Dat Ngo & Aman Khan, Arize, 24:46)
- [Evals](https://aietalks.com/talks/evals) (Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI, 3:59:12)
- [Mastering AI Evaluation: From Playground to Production](https://aietalks.com/talks/mastering-ai-evaluation-from-playground-to-production) (Doug & Carlos Essan, Braintrust, 1:25:08)
- [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)
- [Practical Tactics to Build Reliable AI Apps](https://aietalks.com/talks/practical-tactics-to-build-reliable-ai-apps) (Dmitry Kuchin, Multinear, 14:55)