Evals show whether an AI system works for its real-world application, beyond improvements to model metrics.
2
A good eval setup lets teams run much of the product iteration loop before deploying to production.
3
Using the same metrics on offline tests and production data helps identify useful examples for the next iteration.
Summary
Manu Goyal introduces evals through his experience building autonomous systems at Nuro. In self-driving cars, improving an image classification score was not enough to justify putting a model on the road. The team also needed to know whether it avoided pedestrians, handled traffic situations, and followed the law. Goyal argues that evals provide this application-specific evidence. They are more than unit tests or regression checks. With enough eval coverage, a team can experiment before deployment instead of relying on risky and expensive production feedback. Applying the same metrics to production data can then show which examples deserve attention in the next development cycle. Goyal also describes Braintrust's broader development platform, which connects evals with prompt experiments, playground work, logging, and observability. His conclusion is direct: evals are a way to make AI development faster and more confident because they give teams a laboratory for testing changes.
Model improvements do not prove that an AI system is ready for production
Goyal draws on his work in self-driving cars, where teams could spend time tuning the model, changing its architecture, and adjusting the loss function. A better image classification rate still did not justify putting the system on the road. The relevant questions were tied to the application: whether the car avoided pedestrians, negotiated traffic scenarios appropriately, and obeyed the law. Evals provide a way to connect model behavior with those real-world requirements.
Evals provide application-specific evidence about whether a system works
The point of an eval is to understand how an AI system performs in the setting where people will use it. Goyal contrasts this with relying on a general model score. For an autonomous vehicle, the team needs evidence about behavior in traffic and around pedestrians. The same principle applies to other AI products: teams need tests that reflect the outcomes their product must achieve, rather than metrics that only describe an isolated part of the model.
Evals let teams learn before they ship to production
Goyal says evals are not only unit tests for AI and are not limited to finding regressions. Without them, the only signal from a change might come after shipping it to production. That feedback is expensive, slow, and risky. Investing in evals creates what he calls a laboratory, where a team can run experiments and complete much of the product iteration loop before deployment. This lets the team ship more quickly and with more confidence.
Production data can guide the next eval and development cycle
Goyal describes applying the same metrics used offline to online production data. This creates data-driven information about which production examples will be most useful in the next iteration. The eval system therefore does more than approve or reject a change before launch. It can also help select real examples for future testing and development, connecting what happens in production with what the team studies offline.
Braintrust connects evals with the rest of AI development
At Braintrust, Goyal says the goal is to build a development platform around evals and the work that accompanies them. The described workflow includes tweaking prompts, experimenting in a playground, logging data, and adding observability. Connecting these activities creates what he calls a data flywheel. The intended result is a shared development loop in which experiments, logged behavior, and eval results inform one another.
Goyal presents evals as a practical requirement for AI development
Goyal points to support for evals from Kevin Weil, Gary Tan, Mike Krieger, and Greg Brockman, while keeping the main argument grounded in his own experience. His conclusion is that evals give AI teams a way to test changes before they rely on production outcomes. The talk is an introduction to the evals track, with the practical case centered on reducing the cost and risk of learning after deployment.
"If I didn't have evals, the only way I can get any signal on my changes is by shipping it to prod and then getting signal, you know, in the real world."02:18
Who should watch
You are building an AI product and need evidence that a model or prompt change works for the actual application before deploying it.
Your team relies on production feedback because it has no repeatable way to test behavior offline.
You want to connect prompt experiments, logging, observability, and production examples into one eval workflow.
