# How We Took an AI Agent From 41% to 97% Accurate

Milind Sunil Shah, Brainbase Labs | AI Engineer NYC 2026 | 13:23

Source: https://www.youtube.com/watch?v=4VmugsE-iP0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-we-took-an-ai-agent-from-41-to-97-accurate
Published: 2026-11-05
Tags: cost, evals, human-in-the-loop, prompt-engineering

## TL;DR
- A narrow first version gave the team a measurable baseline, even though it was only 41% accurate and took more than two hours to run.
- Definitions, benchmark cases, evals, and targeted instruction changes raised accuracy to 97% without changing the model.
- The customer took over the full operating loop, including reviewing exceptions, adding benchmark cases, and deciding when a new version was ready.

## Summary
Milind Sunil Shah explains how Brainbase Labs improved an agent that collected competitor pizza prices for Domino's. The agent had to select the right store, interpret delivery and takeaway prices, apply promotions, and write results into a spreadsheet. Version one was 41% accurate, cost $2 per source, and took 129 minutes per run. Version 11 reached 97% accuracy, cost $1 per source, and took 71 minutes. The team did this through a six-step process: agree on the desired outcome, define every term and rule, build benchmarks and evals, ship a narrow first agent, score and fix failures, then hand the operating loop to the customer. Shah gives concrete fixes, such as checking an entire page for deals and reading the HTML DOM before zooming into promotional images. The customer now runs the agent, reviews exceptions, and turns corrections into new benchmark cases.

## Key ideas
### Agree on the work, the score, and the human handoff before building
[02:23](https://www.youtube.com/watch?v=4VmugsE-iP0&t=143s)
The kickoff defines what the human does today, what the agent must produce, how long the work takes, what it may cost, and who judges the result. For Domino's, analysts spent two hours each day creating a spreadsheet, while the pricing team analyzed it. The agent had to collect local competitor prices faster and repeatedly without removing human judgment from edge cases. The team also agreed that the analyst would sign off on the output and that uncertain results would be flagged. Accuracy, source cost, and run time became the measures for the project.

### Give every term a definition so the agent has fewer chances to guess
[03:35](https://www.youtube.com/watch?v=4VmugsE-iP0&t=215s)
The ontology records what the agent collects, where each answer comes from, where it goes, and what to do when the answer is unclear. Shah says every term should ideally have a definition because an unwritten rule becomes a guess. In the Domino's example, a price observation stores the item ID, observed price, full price, and source. The rules distinguish delivery from takeaway, define a large pizza, require promotion math before entering a value, and tell the agent to record a gap instead of guessing. The output can point to the exact page and rule behind a selected price.

### Use old analyst work and captured browser states to build repeatable evaluation
[05:43](https://www.youtube.com/watch?v=4VmugsE-iP0&t=343s)
The team began with spreadsheets that Domino's analysts had already completed and tried to rerun those cases. Archive.org snapshots worked for some competitor websites, but delivery apps did not serve data in the same way. Brainbase Labs therefore stored browser screenshots from the agent's runs to create an internal benchmark environment. Code checked items such as whether every ID was replaced, whether discount math was correct, and whether a price changed by more than 20% from the previous run. An LM judged item selection and promotion use, while an agent checked whether flagged prices were genuinely problematic.

### A first agent should be small enough to measure from its first run
[07:42](https://www.youtube.com/watch?v=4VmugsE-iP0&t=462s)
Shah recommends shipping the smallest useful slice rather than trying to cover the whole task immediately. Version one is meant to establish accuracy, run time, cost, and failure patterns. Domino's started with one app, one item, and one city. The team then added burgers and takeaway options, another app, and other websites. The initial agent was only 41% correct, took more than two hours, and cost $2 per source. That poor result still gave the team a baseline and a clear set of failures to fix before expanding the scope.

### Fix the most expensive or frequent failures with one change at a time
[08:49](https://www.youtube.com/watch?v=4VmugsE-iP0&t=529s)
The iteration loop is score, fix, and ship. The team prioritizes failures by frequency and cost, then checks whether a fix makes run time or spending worse. Shah recommends changing instructions before changing the model and adding useful failures to the benchmark. One Domino's fix told the agent to inspect the whole page instead of stopping at a linked section, because a cheaper deal appeared below the pizza it had found. Another fix told it to inspect the HTML DOM and image titles before zooming into promotional images, and to stop after a set zoom level. That reduced image reads to one third and cut their cost by more than 50%.

### The customer needs the complete correction loop, not only a deployed agent
[11:56](https://www.youtube.com/watch?v=4VmugsE-iP0&t=716s)
The handoff covers who runs the agent, who reviews its output, who turns corrections into rules and benchmark cases, and who decides when a version is ready. Domino's runs the agent on a schedule or through an email trigger. Analysts review exceptions instead of checking every result. Corrections can be added as benchmark cases, and the customer can make small changes themselves. This arrangement lets the team expand the work into more markets and other areas while keeping the review and improvement process explicit.

## Notable quotes
- "Version one's job is to be measured, it's not to be good." (07:54)
- "If you don't write something down, the agent's going to guess and probably get something wrong." (03:55)
- "Ideally, change one thing at a time and fix the instructions before you want to change the model entirely." (09:12)
- "You're trying to hand off an entire loop and not just the agent." (12:08)

## Tools & references mentioned
- Domino's
- Brainbase Labs
- archive.org
- Postgres

## Who should watch
- You are building an agent for messy websites or delivery apps and need a practical way to define correctness.
- Your first version works inconsistently, but you need to decide whether to improve its instructions, its evals, or its scope.
- A customer will operate the agent after launch and needs a clear process for reviewing exceptions and feeding corrections back into development.

## Editor's note

Milind Sunil Shah says the first Domino's agent was only 41% correct, so the team needed a measurable baseline and repeatable failure cases before expanding it. Kitaru records real agent runs, including model calls and tool results, then replays them with the same inputs and responses. That lets a team rerun a failed case after changing the prompt, model, or code.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [How to Test AI Agents When No Benchmark Exists](https://aietalks.com/talks/how-to-test-ai-agents-when-no-benchmark-exists) (Nathanael Weill, Air Canada, 19:50)
- [Why 80% Reliability Isn't Good Enough](https://aietalks.com/talks/why-80-reliability-isnt-good-enough) (Felipe Blanes, Amazon AGI Lab, 17:45)
- [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)
- [We Kept Rebuilding Our AI Agent. Here's What Survived](https://aietalks.com/talks/we-kept-rebuilding-our-ai-agent-heres-what-survived) (Fuad Ali, Arize AI, 20:06)
- [Don't Trust Your AI Agents. Test Them Like a Scientist](https://aietalks.com/talks/dont-trust-your-ai-agents-test-them-like-a-scientist) (Stephanie Jarmak, Omni Analytics, 17:12)
