# Don't Approve the AI Model. Approve the App

Doneyli De Jesus, Langfuse | AI Engineer NYC 2026 | 19:20

Source: https://www.youtube.com/watch?v=-uBABM2AuFg
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/dont-approve-the-ai-model-approve-the-app
Published: 2026-11-04
Tags: evals, guardrails, human-in-the-loop, observability, tracing

## TL;DR
- Regulated banks need to certify the complete AI application, including its prompts, models, tools, and workflow, rather than rely on public model benchmarks.
- A golden data set, trace-based evaluation, and separate gates for different quality dimensions can show whether an application is ready for production.
- Adding a calculator tool raised numerical accuracy from 60% to over 90%, allowing the application to clear the bank's 85% gate.

## Summary
Doneyli De Jesus describes work with a top-five bank that needed deployment controls for AI agents. Public model benchmarks were not useful because they did not test the bank's private data or internal use cases. The bank wanted applications to run on its premises, behind gates, and in air-gapped environments. The proposed process starts with a golden data set and produces traces for every evaluated agent run. Those traces are scored for several dimensions, including grounding, numerical accuracy, the path taken by the agent, and regulatory compliance. A small model scored 97% on groundedness but only 60% on numerical accuracy. Giving it a calculator tool raised numerical accuracy to over 90%, and the application passed an 85% gate. Domain experts set thresholds, inspect failures through annotation queues, and add reviewed examples to the data set. De Jesus says this reduced evaluation work from weeks to minutes while preserving evidence for every configuration change.

## Key ideas
### Private use cases matter more than public benchmarks
[00:32](https://www.youtube.com/watch?v=-uBABM2AuFg&t=32s)
De Jesus says regulated industries may not care about public model benchmarks because those tests do not necessarily predict performance on internal data. Banks treat their data, use cases, and private data sets as sensitive. His example is a 10K analyst for investment bankers. Summarizing an earnings call requires firm-specific knowledge and context that a model may not have out of the box. The bank therefore needs to test the application against its own work rather than infer quality from a public leaderboard.

### Banks certify applications and require controlled deployments
[03:01](https://www.youtube.com/watch?v=-uBABM2AuFg&t=181s)
The bank told De Jesus, "We don't certify models here. We certify applications." An application may be a 10K analyst or an investment analysis tool. The deployment also has to meet the bank's operating constraints. The bank wanted systems on its own premises, with models and data kept under its control, and deployments that were air-gapped. Some banks are rebuilding local GPU infrastructure and training or fine-tuning models themselves because of the sensitivity of the information involved.

### A golden data set provides the application's test cases
[05:00](https://www.youtube.com/watch?v=-uBABM2AuFg&t=300s)
The evaluation process starts with a golden data set. De Jesus compares it to software test cases that should run before anything reaches production. Running the application against the set generates traces, which provide evidence of what the agent did during evaluation. Scores can then be attached to those traces across selected dimensions. A change to a system prompt, model, or tool can trigger a new run and produce a pass or fail result for the application.

### A calculator tool fixes a model's numerical weakness
[07:20](https://www.youtube.com/watch?v=-uBABM2AuFg&t=440s)
The bank's model achieved 97% groundedness because it extracted information from internal sources and based responses on the bank's data. It scored only 60% on numerical accuracy because language models are poor at doing math. De Jesus's team gave the agent a calculator tool for calculations. The tool raised numerical accuracy to about 95%, according to the example shown, and later he describes the result as over 90%. The agent no longer had to perform deterministic arithmetic itself.

### The gate evaluates the whole workflow across separate dimensions
[08:52](https://www.youtube.com/watch?v=-uBABM2AuFg&t=532s)
The agent's workflow included planning, retrieval from official sources, and a calculation step. The evaluation covered more than one overall average. Different dimensions can have different thresholds for accuracy, regulatory compliance, and other requirements. A result can look good overall and still fail because one dimension is unacceptable. De Jesus gives regulatory compliance as an example where a zero can stop promotion entirely. The gate applies to the application and its workflow, so the controls can remain in place if the team changes the model.

### A useful evaluation combines code, path checks, and an LLM judge
[11:32](https://www.youtube.com/watch?v=-uBABM2AuFg&t=692s)
De Jesus describes three scoring approaches. Deterministic code scores the same input consistently and is useful for checks that have a clear expected result. Path evaluation checks whether the agent took the right route, since a plausible answer can come from an inappropriate or incomplete sequence of steps. An LLM judge handles more qualitative aspects of the response. He argues that complicated systems need a blend of deterministic and non-deterministic evaluators instead of relying entirely on LLM judges.

### Domain experts set thresholds and review failure traces
[13:48](https://www.youtube.com/watch?v=-uBABM2AuFg&t=828s)
Technical teams should not set every acceptance bar alone. Risk and other domain experts need to decide what is acceptable for each use case, especially when an application may be used internally or in front of customers. The team can create annotation queues for specific failure modes, such as regulatory compliance failures. Experts inspect the relevant traces and give feedback. Reviewed failures can become new examples in the golden data set, making later evaluations more representative.

### Configuration changes trigger repeatable gates
[15:53](https://www.youtube.com/watch?v=-uBABM2AuFg&t=953s)
When a prompt or other agent configuration is promoted, the gate clearance process runs again. The traces link evaluation results to changes in the system prompt, model, and tools. Human feedback can also produce a better example for a particular evaluation dimension and send it back into the golden data set. De Jesus says this process took application evaluation from weeks to minutes, although regulated risk and compliance work can otherwise take months.

## Notable quotes
- "We don't certify models here. We certify applications." (03:01)
- "The first thing that we need is to build a golden data set." (05:00)
- "You need to write your own benchmark because the public ones don't really matter for your data." (18:00)
- "You have to gate the application, not the model." (18:18)

## Tools & references mentioned
- Langfuse
- ClickHouse
- Opus 5
- Python

## Who should watch
- You are deploying AI agents in a bank or another regulated environment and need evidence that a configuration change is safe to release.
- Your team relies on public model benchmarks, but your application uses private data, internal rules, or domain-specific workflows.
- You need a practical evaluation loop that combines automated checks with reviews from risk or other domain experts.

## Editor's note

Doneyli De Jesus shows how a calculator tool raised numerical accuracy from 60% to over 90%, allowing the application to clear the bank's 85% gate. Kitaru records each real agent run with its model calls and tool results, then replays it with the same inputs and responses after a prompt, model, or tool change. This lets teams check whether a changed application still passes the run that exposed the problem.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Deploying AI Inside a Bank Is Not an Engineering Problem](https://aietalks.com/talks/deploying-ai-inside-a-bank-is-not-an-engineering-problem) (Paden Gayle, Applied AI Engineers, 15:07)
- [Building the AI Agent Is Easy. Proving It's Safe Isn't](https://aietalks.com/talks/building-the-ai-agent-is-easy-proving-its-safe-isnt) (Jonathan Osborne, NatWest, 18:01)
- [If Your AI Scores 100%, Your Eval Is Wrong](https://aietalks.com/talks/if-your-ai-scores-100-your-eval-is-wrong) (Ramakrishnan Lokanathan, eSlams, 18:45)
- [Practical Tactics to Build Reliable AI Apps](https://aietalks.com/talks/practical-tactics-to-build-reliable-ai-apps) (Dmitry Kuchin, Multinear, 14:55)
- [Don't Trust Your AI Agents. Test Them Like a Scientist](https://aietalks.com/talks/dont-trust-your-ai-agents-test-them-like-a-scientist) (Stephanie Jarmak, Omni Analytics, 17:12)
