Evals for High-Stakes AI Agents

Doneyli De Jesus, Langfuse, Annabell Schäfer, Langfuse53:32 · Oct 2026 · 9,015 views
Thumbnail for Evals for High-Stakes AI Agents Watch on YouTube
TL;DR
  1. 1

    AI agents can report success while failing to perform the underlying action, so teams need traces, datasets, and evaluations that expose those failures.

  2. 2

    Domain experts should review representative production traces, while LLMs can group their comments into recurring failure modes for future evaluation.

  3. 3

    Prompt and configuration changes can trigger CI experiments that block deployment when evaluator scores fall below defined thresholds.

Summary

Annabell Schäfer and Doneyli De Jesus present a practical evaluation loop for customer-facing AI agents in regulated settings. They use a banking assistant to show how tracing captures model calls, tool calls, inputs, outputs, scores, and metadata while masking personally identifiable information before persistence. Production traces become curated datasets containing questions, expected outputs, intents, and desired behaviors. Domain experts review a representative sample, add short comments, and help establish ground truth. An LLM can then group those comments into recurring failure modes, which become evaluation dimensions. The speakers recommend combining LLM-as-a-judge evaluators with deterministic code checks. Experiments compare prompts, models, context, and tools against the same dataset. A prompt change can launch a GitHub CI job, run the evaluations, and apply minimum score thresholds before production. Dashboards help business stakeholders inspect automation, cost, user feedback, and risk categories. The talk also sets limits on coding-agent automation: agents can automate instrumentation and analysis, but humans must define the desired behavior and hard constraints.

Key ideas
02:09

Green system metrics can hide an agent that lied about completing an action

Annabell Schäfer opens with a support agent that tells a user it issued a refund. Latency and traditional system status remain green, but the refund tool was never called. The team might discover the failure only weeks later when the user reports that no refund arrived. She uses this example to explain why nondeterministic systems need continuous ways to find failures and prevent their recurrence. Logs and uptime checks do not establish that the agent actually did what it claimed.

04:39

Tracing provides the inspectable record needed for evaluation

The AI engineering loop starts with tracing every step of an agent execution. A trace can include the root operation, aggregate time and cost, individual model generations, tool calls, inputs, outputs, and metadata. This record supports cost and latency analysis, human investigation, and automated inspection by coding agents. Monitoring then aggregates those traces to find interesting cases, including compliance failures, frustrated users, and other quality signals that are difficult to find by reading every log manually.

07:42

Datasets turn desired agent behavior into repeatable test cases

Datasets contain representative examples of what an agent should handle, including out-of-scope requests that it should defer. Each item can include an input, a sample reply, expected behavior, or required keywords. Doneyli De Jesus compares these items with software test cases. They provide a stable benchmark for offline iteration and CI/CD. A dataset can focus on a narrow concern, such as prompt injection, unresolved disputes, or whether the agent respects a particular banking rule.

12:38

PII masking and guardrail scores make banking traces usable

The banking assistant identifies and masks personally identifiable information before it is persisted in Langfuse. When a user enters a credit card number, the trace retains a redacted value rather than the number in plain text. The demo also scores whether answers comply with banking boundaries and whether guardrails catch risky intents. A request for investment advice is flagged so the team can filter for those cases and verify that the assistant does not provide advice it should defer.

24:02

Expert comments can become a library of failure modes

Doneyli recommends sampling production traces for human review instead of asking experts to inspect thousands of interactions. An annotation queue can ask a domain expert for a few simple scores and a comment, such as 'Citation is not there. Needs to be included in all these cases.' An LLM can analyze the accumulated comments and extract recurring categories. Those categories become measurable dimensions such as source correctness, formal register, language matching, and required content.

30:24

A blended evaluator is safer than relying only on an LLM judge

The speakers recommend mixing LLM-as-a-judge evaluators with deterministic code evaluators. An LLM judge can assess qualities such as correctness or whether a response follows a desired behavior, but its output can vary. A code evaluator gives the same result for the same input, such as an exact-match or regular-expression check. Combining both produces a broader assessment while preserving checks that do not depend on another model's judgment.

34:00

CI can block a prompt change when the agent regresses

An experiment runs a configured agent against a dataset while changing one part of the setup, such as the system prompt, model, context, or tools. The demo connects prompt changes to a GitHub CI job. When a new prompt version is promoted, the job runs the evaluations and compares scores with minimum thresholds. A failed threshold means the configuration is not considered a pass. The resulting evidence gives the team a record for its deployment decision instead of relying on an informal impression that the agent looks good.

46:06

Coding agents should automate repetitive work while humans set the standard

Annabell Schäfer rejects both extremes of fully manual evaluation and an instruction such as 'fix my agent' with no constraints. Coding agents can automate instrumentation, failure-mode discovery, evaluation runs, and repeated experiments. Domain experts still need to define what the system should do, curate datasets, and set hard limits. In financial services, one such limit is that the assistant must not give investment advice. The proposed balance keeps human judgment in the loop where the desired behavior is defined.

"You should get your domain experts in the room to build out the data sets to curate what is actually the problem that your application should be solving."Annabell Schäfer47:14
Who should watch
  • You are shipping a customer-facing agent in banking, healthcare, or another regulated setting and need evidence that it follows its boundaries.
  • Your team has production traces and expert feedback but no repeatable dataset or evaluation gate for prompt and model changes.
  • You want coding agents to help with eval implementation without handing them control over the definition of acceptable behavior.