Nubank uses simulated conversations to replace much of the waiting involved in collecting production data for agent evaluations.
2
Simulations generate grounded, multi-turn interactions with personas, account context, tool calls, and consistent state, then feed them into existing evaluation pipelines.
3
The approach helped Nubank test agent changes and open models faster while catching regressions and preserving or improving customer satisfaction and self-service rates.
Summary
Aman Gupta and Shreya Rajpal describe how Nubank uses Snowglobe simulations to evaluate customer-support agents before exposing changes to live users. Agent evaluation data is difficult because each example is a stateful, multi-turn trajectory with tool calls, account data, and changing state. Manual examples take time to create, while production traces require experiments on real customers and can take a long time to produce reliable feedback. Snowglobe wraps an existing agent, mocks the needed tools, and generates conversations from specified personas and use cases. The resulting data includes grounded context and can run through the team's evaluation pipeline. Nubank reports that human reviewers found simulations usable about 80% of the time, including for new agents. The team uses the simulations to compare versions, catch regressions, reduce unnecessary A/B tests, and try open models. The speakers also stress that teams must measure the gap between simulated and real data before trusting the results.
Evaluation data, rather than metrics, is the main bottleneck
Shreya Rajpal says agent evaluations depend on metrics and data. Teams have increasingly reliable ways to build metrics with language-model judges, human alignment, and prompt optimization. The harder problem is producing the data those metrics evaluate. An agent example is a multi-turn trajectory with internal tool calls and state that must remain consistent. This makes each example more expensive to generate and annotate than a structured machine-learning row or a single-turn question-and-answer pair. Scaling a small set of these trajectories becomes even harder.
Manual data and production traces each impose a different cost
Teams mainly get evaluation data by authoring it manually or collecting production traces. Manual authoring requires people to plan state updates, design the intended trajectory, and keep state consistent across tool calls. Production traces are almost free to collect, but they mean testing changes on live users. That makes large parallel experiments difficult. Aman Gupta adds that changing an agent harness, prompt, tools, or underlying model can take hours, while offline evaluation on hand-curated data can take days. A/B tests can take longer still because customer feedback is sparse and noisy.
Simulation compresses the release cycle from weeks to hours
Gupta says Nubank verified in production that simulations can reduce the evaluation timeline from a few weeks to less than a day, and sometimes to a few hours or minutes. The agent harness still takes time to change, but evaluation no longer waits for people to curate examples. A simulated user can provide data on demand, so the team can get an early signal about whether a new version is likely to work in production. This lets engineers test more ideas before committing to live experiments.
Snowglobe generates grounded, multi-turn conversations around an existing agent
Rajpal explains that Snowglobe wraps an agent through its SDK without requiring code changes. The team specifies which tools need to be mocked, then supplies personas, use cases, and other inputs that steer the simulation. Snowglobe produces thousands of multi-turn conversations against the real agent, with mocked tools and the details needed for evaluation. Judges can then score the conversations per turn and send the results into the existing evaluation pipeline. This gives the team evaluation data when it needs it.
A useful simulation preserves identity, context, intent, and state
The speakers illustrate the output with Maria Souza, a synthetic 34-year-old designer and first-time credit-card customer who wants to order a card. The simulation creates grounding data such as a fake address and fake credit card, then keeps those details consistent throughout the agent's execution. It also specifies how Maria speaks, including her curt, one-line messages. When the agent calls a tool, the simulator returns valid mocked information that can identify the user and verify the address. The conversation therefore follows the intended persona and remains connected to account data.
The team runs a repeated loop of shipping, observation, simulation, and optimization
Gupta describes Nubank's self-improvement loop. The team ships an agent, observes its behavior, creates evaluations, runs simulations, and sends simulated and real data through the evaluation system. The resulting signal is used to optimize the agent harness, including with automated prompt optimization. Once the change is verified, the team ships it and repeats the process. This supports many A/B tests across many agents, while simulations can eliminate weak ideas before they reach production.
Simulation quality must be checked against real conversations
Nubank compares evaluation results from simulated and production data to test whether simulations are useful. The speakers report a high correlation between the two sources and say that human review found simulated data usable about 80% of the time. That finding applied to mature agents and greenfield agents. The team uses simulations to bring up new agents, improve existing ones, and catch problems before deployment. One detected regression would otherwise have reached production, while another issue could have reduced the self-service rate.
Cheap simulation makes model and agent-variant experiments practical
Nubank uses simulations to test open-source models inside its agent harness and compare them with frontier models. Instead of launching every candidate as a live A/B test, the team runs many ideas through simulation and launches only the versions that look promising. Gupta says this saved multiple weeks of effort. The team also reports that simulation helped improve one agent's TNPS by two times and improved self-service rate by 4% in one case. Rajpal's final condition is that teams must establish aligned metrics and measure the simulation-to-real gap before using this loop.
"If you have align metrics that are able to really catch the signals you care about and you have a reliable way of generating data that those metrics can give you signal on, it's then very easy to put together a loop of an agent that continuously improves itself."15:27
Who should watch
You are building a multi-turn agent and hand-curated evaluation conversations are slowing down experiments.
Your team wants to test prompts, tools, harnesses, or models without repeatedly exposing unfinished changes to live users.
You need a way to judge whether synthetic conversations match real customer behavior before using them for release decisions.
