# Building the AI Agent Is Easy. Proving It's Safe Isn't

Jonathan Osborne, NatWest | AI Engineer NYC 2026 | 18:01

Source: https://www.youtube.com/watch?v=5UgnatzpZ9E
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-the-ai-agent-is-easy-proving-its-safe-isnt
Published: 2026-11-04
Tags: evals, guardrails, human-in-the-loop, observability

## TL;DR
- NatWest limits the agent's attack surface with a routing funnel that filters and classifies conversations before they reach generative models.
- A single quality score hid large differences between happy paths, edge cases, and topic changes, so NatWest scores safety, task completion, and customer experience separately.
- LLM judges can be overly generous and poorly aligned with human reviewers, so their models, thresholds, and correlations need continuous checks in production.

## Summary
Jonathan Osborne describes NatWest's move from scripted answers in Cora to AI-generated answers and agents that can take actions for customers. The agent itself was relatively easy to build. The harder problem was proving that it could operate safely across 19 million customers and about 100,000 daily conversations. NatWest reduces risk before generation with a routing funnel, deterministic rules, an intent classifier, and policy checks. It evaluates quality across safety, task completion, and experience rather than using one accuracy number. Human reviews exposed a 54-point gap between happy paths and topic changes. An LLM judge scored much more generously than humans and had almost no correlation with their results. Osborne explains how production traces, sampled evaluations, regression gates, shadow deployment, feature flags, and required human feedback support a gradual rollout. The talk is candid about the limits of automated evaluation, especially when the judge itself changes.

## Key ideas
### The risk comes from the agent's blast radius
[00:01](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=1s)
NatWest has about 19 million retail customers, and Cora handles around 100,000 conversations a day. A wrong answer can affect a real customer and real money, bring difficult questions from the regulator, and damage the bank's reputation. Osborne says the model is generally capable of producing good answers. The harder problem is deciding whether it should answer at all, and proving that its behavior is safe at this scale. Cora is moving approximately a third of its journeys from scripted answers to AI-generated answers backed by retrieval, knowledge bases, and agents that can take actions such as making a payment or changing an address.

### Agents add failure modes that scripted systems can bound more easily
[03:40](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=220s)
Osborne names four failures that appear when a system reasons, decides, and generates. It can be confidently wrong, including hallucinating a fictitious source. It can be manipulated through prompt injection. It can give an accurate answer that never actually resolves the customer's problem. It can also be nondeterministic, producing different outputs for the same input. Existing regression tests and software assertions do not cover this behavior as well as they cover a scripted bot, whose possible responses can be written out and tested.

### A routing funnel keeps many conversations away from generative models
[04:52](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=292s)
NatWest first shrinks the attack surface. A contextual filter removes common events such as button clicks, which Osborne says account for about 20%. Deterministic pre-classification rules then route utterances using the customer's text and metadata without machine learning. Only the remaining conversations reach a gated relevancy and rephrasing step, an intent classifier built as a neural network over embeddings, and policy rules that check whether the customer should enter a particular backend. The design supports access control, auditability, resilience, and configuration changes without a full redeployment.

### Quality needs separate scores for safety, task completion, and experience
[06:34](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=394s)
A 94% intent accuracy score does not answer whether an answer is grounded, whether the customer's problem was resolved, or whether the system knew when to stop and involve a human. NatWest therefore treats quality as a vector with three dimensions: safety, task, and experience. Each can be monitored and alarmed independently. This distinguishes a harmful hallucination from an answer that is technically correct but frustrating to use. Osborne also argues that confidence matters: a router that is highly confident in its wrong decisions can be more dangerous than one that is less confident and knows when it is uncertain.

### Human review exposed a 54-point gap hidden by an 83% pass rate
[07:43](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=463s)
NatWest reviewed 111 payment-agent conversations across 37 payment journeys, with four brands and three independent human scorers per conversation. The overall pass rate was 83%, but the breakdown was more useful: 97% for happy-path scenarios, 76% for edge cases, and only 43% for topic changes. The headline score therefore hid a 54-point spread. Human agreement also depended on the criterion. Reviewers agreed 95% of the time on bias and fairness, but only 64% of the time on whether the bot understood the user's intent. Two of the largest failure causes were engineering issues rather than model problems.

### An LLM judge can score generously while failing to rank cases correctly
[09:16](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=556s)
In an internal complaints agent, around 14,000 cases were scored by both humans and an LLM judge. For faithfulness, the judge's mean was about .97 with little spread, while the human mean was .83 with a much wider spread. The average difference might look acceptable, but the correlation was almost zero. A judge that does not distinguish good cases from bad ones is not useful, and Osborne says LLMs tend to be generous by default. Changing only the judge model caused the measured score to fall from above .8 to about .2, so the judge model itself must be treated as part of the evaluation configuration.

### Production evaluation combines traces, samples, regression gates, and model-specific monitoring
[11:06](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=666s)
NatWest captures live agent and guardrail conversations using open standards such as OpenTelemetry and OpenInference. It samples 5% of conversations by default, while also selecting interesting cases such as handoffs, failures, long conversations, and angry users. The bank scores these with an LLM judge and stores traces, scores, and experiments in Comet. Its environment is air-gapped, and personally identifiable information cannot leave the network. Offline scoring runs every four hours, while prompt and context changes go through CI/CD with a hard regression gate. Generative AI is monitored for regressions, while the traditional intent classifier is monitored for drift and F1 score.

### Shadow mode and feature flags turn approval into smaller decisions
[14:06](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=846s)
NatWest first deploys an agent in shadow mode. It classifies real customer utterances, but the existing scripted system still sends the response. This gives the team production input data without exposing customers to the new behavior. The bank then rolls out by brand and percentage, using KPIs for go or no-go decisions. Some decisions are automated and some need manual approval. Feature flags can send all traffic back to the scripted solution, or isolate each backend by brand or another dimension, and those changes can be made in seconds.

### Required human feedback creates a reusable quality dataset
[15:42](https://www.youtube.com/watch?v=5UgnatzpZ9E&t=942s)
For the complaints agent used by complaint handlers, human feedback is a gate rather than a thumbs-up control. A handler who wants to rerun an investigation must explain why the rerun is needed. That requirement has produced about 14,000 colleague labels, which Osborne describes as valuable because other projects often struggle to collect even a tenth of that amount. The data supports judge calibration and regression checks. His broader conclusion is that the reusable evaluation and monitoring layer takes more work than the agent code, especially when the agent is customer-facing and regulated.

## Notable quotes
- "The actual creation of the agent that's kind of the easy part. The much harder part I think is proving that it's safe." (00:01)
- "If we build an agent that can answer any question, it can also answer any question wrong." (03:28)
- "You can't embody that with one single score." (06:56)
- "A judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless." (10:04)
- "Continuous evaluation is part of your production infrastructure. It's not a notebook that you run a couple of times when your model goes live." (12:26)

## Tools & references mentioned
- NatWest Group
- Cora
- Comet
- OpenTelemetry
- OpenInference
- AWS

## Who should watch
- You are putting an AI agent in front of customers and need evidence for risk, compliance, or model governance approval.
- Your evaluation dashboard has one headline accuracy number, but you do not know how performance changes on edge cases or topic changes.
- You need a rollout plan that can collect production data without making the new agent answer customers immediately.

## Editor's note

Jonathan Osborne shows that a single evaluation score can hide serious failures, including a 54-point gap between happy paths and topic changes. Kitaru records each real agent run with its model calls and tool results, then replays it after a prompt, model, or code change. That gives teams a repeatable way to check whether a previously observed failure still occurs or a fix breaks the recorded run.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Don't Approve the AI Model. Approve the App](https://aietalks.com/talks/dont-approve-the-ai-model-approve-the-app) (Doneyli De Jesus, Langfuse, 19:20)
- [What a $900M Bank Error Teaches Us About AI Agents](https://aietalks.com/talks/what-a-900m-bank-error-teaches-us-about-ai-agents) (John Feminella, Two Sigma, 18:56)
- [Deploying AI Inside a Bank Is Not an Engineering Problem](https://aietalks.com/talks/deploying-ai-inside-a-bank-is-not-an-engineering-problem) (Paden Gayle, Applied AI Engineers, 15:07)
- [Scaling Agents for Gen AI Products](https://aietalks.com/talks/scaling-agents-for-gen-ai-products) (Anju Kambadur, Bloomberg, 19:38)
- [Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait](https://aietalks.com/talks/shipping-ai-into-5-major-banks-without-a-12-month-compliance-wait) (Al Chen, Cisco, 17:42)
