Agents can drift away from their requirements as models, prompts, environments, and edge cases change, so teams need continuous tracing, evaluation, and monitoring.
2
Trace-linked evaluations shorten the path from detecting an agent failure to finding the part of the workflow that caused it.
3
Microsoft Foundry's early-preview observe skill can generate evaluation data, run batch evaluations, optimize prompts, compare versions, and restore the best version with human guidance.
Summary
Amy Boyd and Nitya Narasimhan explain observability for agents through the idea of a widening gap between an agent's requirements and its behavior. They cover Microsoft Foundry's portal and SDK workflows for building agents, adding tools, tracing calls, evaluating quality and safety, and inspecting multi-agent workflows. Nitya argues that evaluations become much more useful when linked to traces, because developers can move from a failed metric to the specific tool call or agent step that caused it. The talk also covers red teaming, where an AI generates adversarial prompts to test guardrails. The main demonstration uses Foundry's early-preview observe skill through GitHub Copilot. Starting with only an agent endpoint and instructions, it creates an evaluation dataset, runs a baseline, diagnoses failures, optimizes the prompt, evaluates new versions, and rolls back to the best result. The skill's reasoning gives developers evidence about failures they had not anticipated.
Agent observability is about measuring the gap between requirements and behavior
Amy Boyd compares an agent with a train and its requirements with a station platform. A new train may fit the platform, while changes in the agent can create a wide gap. She recommends running evaluations from the earliest build stage and throughout the agent's life cycle so requirements stay connected to what the agent does in production. The analogy also includes safety guardrails and the need to keep watching agents as platforms, trains, customers, and environments change. The goal is to know what the agent is doing today and whether it continues to meet expectations over time.
Reliability requires evaluation, monitoring, and optimization
Amy describes three parts of managing agent non-determinism. Evaluation measures performance, quality, safety, and how well the agent is doing its task. Monitoring continues after development as requirements, customers, and environments change, helping teams debug problems quickly. Optimization uses the data from those evaluations and traces to decide what to change. Foundry supports these activities across three phases: early tracing while building, debugging and optimization in production, and eventually centralized oversight across many agents and multi-agent systems.
Evaluations should inspect intent, tool calls, and final task completion
For a weather agent, Nitya says evaluation can happen at several points in the workflow. Intent resolution checks whether the agent understood that the user wanted local weather. Tool-call evaluation checks whether it selected the expected tool, often using percentages because agent behavior is non-deterministic. The final response can be evaluated for task completion and task adherence. This breakdown lets developers locate the stage where quality drops instead of judging only the final answer. Foundry also provides built-in quality, safety, and agent-specific evaluators, while custom evaluators can cover scenarios the built-ins do not fit.
Trace-linked evaluations connect a failed score to its cause
Nitya says it is not enough to know that something went wrong. Developers also need to shorten the time between detection and diagnosis. Linked evaluations let a developer start with a failed metric, return to the corresponding trace, and compare the behavior with an earlier version. For example, after changing a model, a tool-call metric might fail because one tool was no longer called efficiently. Traces can show what changed in the workflow, while evaluation results show the effect of that change. This connection applies regardless of how the agent was built, as long as it exposes an endpoint and emits the required telemetry.
Multi-agent workflows make tracing more useful as complexity grows
Nitya builds a travel application from a single agent with functions into a workflow with dedicated flight, hotel, and car agents. A concierge workflow orchestrates those specialists through a declarative YAML definition. The resulting trace shows how the request moved through each agent and how their responses were combined. This lets a team identify which specialist is underperforming, inspect token costs, and optimize only the part that needs work. Foundry can trace the workflow without requiring an external agent framework, and the same approach can be used with agents built elsewhere if they emit OpenTelemetry traces.
Red teaming tests whether guardrails survive manipulated prompts
Red teaming uses a second AI to attack the target agent with prompts from selected risk categories. Nitya gives an example where a direct request to rob a bank is blocked, but a manipulated prompt using reversed text might pass a superficial guardrail before the model reverses it and follows the harmful request. The red-team agent generates attacks and reports the cases where the target failed. Tests can cover prohibited actions such as leaking data or filling in passwords. Attack strategies range from simple leetspeak to crescendo attacks, which build from harmless prompts into a more serious request over several turns.
The observe skill can create an evaluation loop from almost no setup
Nitya demonstrates the observe skill through GitHub Copilot with an agent that has instructions, a model, and an endpoint but no evaluation dataset or baseline. The skill inspects the project, creates an evaluation dataset, selects evaluators, and runs a first batch evaluation. It finds that relevance is acceptable while task adherence has failures, then explains the failures and asks whether it should continue. When told to proceed, it optimizes the instructions, creates a new agent version, reruns the evaluation, and reports the comparison. The process can try further changes, compare regressions, and restore the version with the best result.
Human guidance remains part of automated agent optimization
The observe skill does not remove the developer from the loop. Nitya shows it trying multiple prompt changes, with quality improving and then regressing across versions. The developer can tell it to stop and select the best version from the history. The skill explains what worked and what did not, exposing failures and possible causes that the developer might not have considered. Nitya says it can also investigate other changes, such as whether web search is taking too much time, and can be guided with domain knowledge. The automation handles repeated evaluation and comparison, while the human decides what direction is acceptable.
