Voice AI Where One Wrong Question Breaks the Rules

Shiv Indap, Upstart, Brooke Hopkins, Coval17:03 · Oct 2026 · 1,676 views
Thumbnail for Voice AI Where One Wrong Question Breaks the Rules Watch on YouTube
TL;DR
  1. 1

    Upstart's loan verification agent must follow different questioning rules depending on what an applicant says, because a wrong question can create a regulatory problem.

  2. 2

    Human review exposed gaps in both the LLM judge and the agent, raising agreement between the judge and reviewers from 33% to 92%.

  3. 3

    The improvement loop monitors calls, routes failures to the right reviewers, tests fixes in voice simulations, and keeps human approval before production changes.

Summary

Shiv Indap and Brooke Hopkins describe how Upstart evaluates a voice agent that handles loan verification calls. The agent asks about continuing income, use of funds, and outstanding obligations, but compliance rules change based on the applicant's answers. For example, asking the amount of money needed for medical use is prohibited. Upstart first reviewed every call in a Google Sheet, which quickly became difficult to manage. It then introduced structured human review and automated LLM judges. Reviewers compared judge outputs with expert decisions, found missing tribal knowledge and judge errors, and helped raise agreement from 33% to 92%. Hopkins presents the wider loop: define metrics from what reviewers care about, monitor live calls, route failures to appropriate teams, and re-simulate fixes with voice interactions before deployment. Self-improving agents may suggest or automate fixes, but engineers or compliance staff still approve changes.

Key ideas
01:07

Regulated voice calls need rules that depend on the applicant's answer

Upstart handles thousands of calls each day, and its lending domain makes mistakes more serious than ordinary customer experience problems. The verification agent asks whether income will continue for three months, how the borrower plans to use the funds, and whether there are outstanding obligations. It must first verify the caller's identity. The next question depends on the answer. If the borrower says the funds are for medical purposes, the agent cannot ask for the amount. If the borrower mentions travel to a graduation, the agent needs to establish whose graduation it is. The agent also avoids answering questions about application status or when money will arrive.

04:07

A shared spreadsheet cannot support growing human review

Upstart required a human to review every call after launch. The operations team began with a Google Sheet, but the review process quickly became disorganized. Five columns became ten, then fifteen, as people added fields and entered information in different ways. The goal was to collect data that could improve both the agent and the performance criteria. That required a more structured review process, which Coval helped build.

05:28

Metrics should come from the behavior the team needs to act on

The team initially tracked many measures, including call time, accuracy, and clarity. It later reduced the list to measures that mattered most for the use case. Examples included accuracy, clarity, completeness, policy alignment, correct escalation, and CSAT. Brooke Hopkins says teams should start with sensible metrics and discard them when they stop reflecting what matters. Human reviewers first label the behavior they care about, and those labels can then become automated evaluations. Starting with a measure such as word error rate is unhelpful if the team does not take action based on it.

07:11

Human feedback improved both the judge and the agent

For compliance, an LLM judge scores each completed call and sends its result to a queue for human review. Reviewers explain whether the judge's decision is correct. At launch, the judge and human reviewers agreed only 33% of the time. The team found that important tribal knowledge had not been written into the agent. Reviewers also disagreed with one another because some guidance was unwritten. Encoding that knowledge and improving the judge across more use cases raised agreement to 92%. The number is treated as something to improve rather than something expected to reach 100%.

10:04

Live monitoring should focus human attention on failures

Coval groups evaluations into whether the agent reached the intended outcome, followed the required steps, and delivered an acceptable conversation experience. These evaluations can use LLM judges, heuristics, and audio processing. They monitor live calls for known failures such as missing loan verification or incorrect wording. Clustering and anomaly detection can find failures the team did not anticipate. Review queues then send relevant calls to human reviewers instead of making them inspect every call. Failures can go to different teams, such as the agent team for off-script behavior or compliance for regulatory violations.

12:31

Voice simulations test a fix before users encounter it

Once reviewers identify real failures, the team can determine a change and re-simulate it. Coval's simulations call the agent and evaluate voice interactions with background noise, different accents, voice quality, and interruptions. This lets the team check whether a fix actually addresses the problem before releasing it. Hopkins describes this as shifting failure detection toward development rather than waiting for production. The aim is to avoid deploying a suspected fix and discovering its effects through live users.

14:04

Self-improvement still requires human approval

The proposed self-healing loop can examine evaluations, identify fixes, and use reinforcement learning pipelines or fine-tuned language models to improve the system. Those changes are gated by human review. Engineers usually approve judge-related changes, while compliance staff approve changes that affect compliance behavior. Feature flags also control changes. In a fintech setting, self-healing does not mean the system changes its production behavior without someone signing off.

"We want to shift left to use a DevOps term. We want to detect as many failures as we can in simulation instead of doing that in production."Brooke Hopkins13:13
Who should watch
  • You are building voice agents for lending, insurance, healthcare, or another regulated setting and need a concrete review process.
  • Your team has outgrown ad hoc spreadsheets for evaluating calls and wants to connect human labels to automated judges.
  • You are considering self-improving agents and need a release process that tests changes before production while retaining human approval.