A narrow first version gave the team a measurable baseline, even though it was only 41% accurate and took more than two hours to run.
2
Definitions, benchmark cases, evals, and targeted instruction changes raised accuracy to 97% without changing the model.
3
The customer took over the full operating loop, including reviewing exceptions, adding benchmark cases, and deciding when a new version was ready.
Summary
Milind Sunil Shah explains how Brainbase Labs improved an agent that collected competitor pizza prices for Domino's. The agent had to select the right store, interpret delivery and takeaway prices, apply promotions, and write results into a spreadsheet. Version one was 41% accurate, cost $2 per source, and took 129 minutes per run. Version 11 reached 97% accuracy, cost $1 per source, and took 71 minutes. The team did this through a six-step process: agree on the desired outcome, define every term and rule, build benchmarks and evals, ship a narrow first agent, score and fix failures, then hand the operating loop to the customer. Shah gives concrete fixes, such as checking an entire page for deals and reading the HTML DOM before zooming into promotional images. The customer now runs the agent, reviews exceptions, and turns corrections into new benchmark cases.
Agree on the work, the score, and the human handoff before building
The kickoff defines what the human does today, what the agent must produce, how long the work takes, what it may cost, and who judges the result. For Domino's, analysts spent two hours each day creating a spreadsheet, while the pricing team analyzed it. The agent had to collect local competitor prices faster and repeatedly without removing human judgment from edge cases. The team also agreed that the analyst would sign off on the output and that uncertain results would be flagged. Accuracy, source cost, and run time became the measures for the project.
Give every term a definition so the agent has fewer chances to guess
The ontology records what the agent collects, where each answer comes from, where it goes, and what to do when the answer is unclear. Shah says every term should ideally have a definition because an unwritten rule becomes a guess. In the Domino's example, a price observation stores the item ID, observed price, full price, and source. The rules distinguish delivery from takeaway, define a large pizza, require promotion math before entering a value, and tell the agent to record a gap instead of guessing. The output can point to the exact page and rule behind a selected price.
Use old analyst work and captured browser states to build repeatable evaluation
The team began with spreadsheets that Domino's analysts had already completed and tried to rerun those cases. Archive.org snapshots worked for some competitor websites, but delivery apps did not serve data in the same way. Brainbase Labs therefore stored browser screenshots from the agent's runs to create an internal benchmark environment. Code checked items such as whether every ID was replaced, whether discount math was correct, and whether a price changed by more than 20% from the previous run. An LM judged item selection and promotion use, while an agent checked whether flagged prices were genuinely problematic.
A first agent should be small enough to measure from its first run
Shah recommends shipping the smallest useful slice rather than trying to cover the whole task immediately. Version one is meant to establish accuracy, run time, cost, and failure patterns. Domino's started with one app, one item, and one city. The team then added burgers and takeaway options, another app, and other websites. The initial agent was only 41% correct, took more than two hours, and cost $2 per source. That poor result still gave the team a baseline and a clear set of failures to fix before expanding the scope.
Fix the most expensive or frequent failures with one change at a time
The iteration loop is score, fix, and ship. The team prioritizes failures by frequency and cost, then checks whether a fix makes run time or spending worse. Shah recommends changing instructions before changing the model and adding useful failures to the benchmark. One Domino's fix told the agent to inspect the whole page instead of stopping at a linked section, because a cheaper deal appeared below the pizza it had found. Another fix told it to inspect the HTML DOM and image titles before zooming into promotional images, and to stop after a set zoom level. That reduced image reads to one third and cut their cost by more than 50%.
The customer needs the complete correction loop, not only a deployed agent
The handoff covers who runs the agent, who reviews its output, who turns corrections into rules and benchmark cases, and who decides when a version is ready. Domino's runs the agent on a schedule or through an email trigger. Analysts review exceptions instead of checking every result. Corrections can be added as benchmark cases, and the customer can make small changes themselves. This arrangement lets the team expand the work into more markets and other areas while keeping the review and improvement process explicit.