# Evals

Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI | AI Engineer World's Fair 2025 | 3:59:12

Source: https://www.youtube.com/watch?v=Vqsfn9rWXR8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/evals
Published: 2025-06-06
Tags: evals, observability, rag, reranking, testing

## TL;DR
- Evals should measure the behavior of the whole AI system, while prompts, tools, code, and models remain replaceable parts.
- Agent teams should turn traces, user feedback, and observed failures into targeted evals, then use trajectory tests and production experiments to check real behavior.
- Good evals are built around a product's actual data and user needs, including retrieval quality, conversation clusters, human perception, and business outcomes.

## Summary
This Evals track brings together several views on testing AI systems. One speaker argues that prompts are a poor programming abstraction because they mix task definitions with model-specific tricks, formatting instructions, and inference strategies. Engineers should separate natural-language specifications, code, tools, and evals so models and optimizers can change independently. Zapier's team describes a data flywheel built from traces, explicit and implicit user feedback, failure clustering, unit-style evals, trajectory evals, and A/B tests. Vercel's Ido Pesal uses a basketball-court metaphor to explain how eval data should cover the real boundaries of an application. Other talks focus on perceptual metrics for generative media, enterprise adoption, fast retrieval evals, and clustering conversations to guide product decisions. Braintrust presents offline and online workflows for tasks, data sets, scores, production tracing, and human review.

## Key ideas
### AI engineering should target the application problem while lower-level models and search methods change
[12:14](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=734s)
The speaker separates maximizing intelligence from building reliable software. Search and learning scale well when the goal is to make an agent learn quickly in a new environment. An application has a different goal, such as controlling information flow, composing functions, or solving a defined user problem. Software is built because general intelligence is unreliable, and engineers need systems that are controllable, scalable, and understandable. The bitter lesson still matters, but it should discourage premature low-level hand-engineering rather than engineering itself. The speaker compares this to premature optimization in software and argues that domain knowledge becomes harmful when it is added below an abstraction level that the engineer cannot justify.

### Prompts mix stable system requirements with temporary model tricks
[21:50](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=1310s)
The speaker calls a prompt a "stringly typed canvas" that entangles several unrelated concerns. It can contain the actual task definition, a formatting request, a model-specific example, an inference-time strategy, and instructions such as producing XML or JSON. These details may have worked for one model at one point, but the prompt gives no clear way to distinguish durable requirements from accidental tweaks. The proposed alternative is separation of concerns. Natural-language descriptions should express what cannot be stated another way. Code should define tools, structure, information flow, and reliable composition. Evals should describe what the system actually needs to preserve while models, prompting methods, and learning algorithms are swapped.

### Agent evals need a data flywheel built from traces and user behavior
[30:06](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=1806s)
Vitor Balogh and Rafal Wilinski explain that shipping an agent changes the team's job from making a prototype work to building a data flywheel. Traces should include tool calls, tool errors, preprocessing, postprocessing, and enough runtime-shaped data to make runs repeatable for evals. Explicit feedback is useful, but users rarely submit it, so teams should also inspect implicit signals such as enabling an agent, copying a response, repeating a request, telling the agent to stop, or abandoning a product. They recommend turning interesting failures into evals with as little friction as possible, then clustering failures to identify the tools and interactions that deserve attention.

### Different eval types answer different questions about an agent
[39:26](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=2366s)
Zapier's team describes a hierarchy resembling the testing pyramid. Unit-style evals check the next state, such as whether the right tool was called or whether its parameters are correct. They are easy to create and useful for hill-climbing specific failures, but they can overfit to one model's path through a task. Trajectory evals let the agent run to an end state and grade the tool calls and artifacts along the way. They are slower and harder when tools have side effects, so Zapier mirrors the user's environment rather than simply mocking it. Rubric-based LLM judges provide a broader view for benchmarking models. Production A/B tests remain the final check because user satisfaction matters more than an isolated eval score.

### An eval data set should map the real boundaries of the product
[54:45](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=3285s)
Ido Pesal uses a basketball court to explain eval design. Each user query is a point, the task is the shot, and the score checks whether the result reaches the basket. Easy queries sit near the basket, difficult but valid queries sit farther away, and out-of-domain requests are outside the court. Teams should avoid spending time on cases users do not care about. They should collect thumbs-up and thumbs-down feedback, inspect random production samples, read community reports, and examine other user signals. The goal is to know where the application succeeds and where it fails, then add data across the whole court instead of concentrating tests around a small set of convenient examples.

### Perceptual AI systems need metrics that account for human judgment
[1:36:23](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=5783s)
Diego Rodriguez argues that generative media evaluation cannot stop at measurements that are easy to calculate. Humans immediately recognize that an AI-generated hand looks wrong, while a model may spend time analyzing the image and still fail to answer naturally. He connects this problem to JPEG and MP3, which remove information that humans perceive poorly while preserving what they notice. Some standard image metrics can change sharply because of JPEG artifacts even when the images look nearly identical to people. For creative systems, evaluation should account for perception, artifacts in training data, and the meaning or opinion conveyed by an image. He suggests training perceptually aware classifiers with examples judged by humans.

### Enterprise evals become valuable when they connect model behavior to business risk
[1:50:59](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=6659s)
John Dickerson frames evaluation and monitoring as two sides of measurement. Generative AI first made executives familiar with the technology, then agentic systems made the consequences more visible because systems began taking actions for people and teams. CFOs need numbers tied to budgets and outcomes. CISOs care about risks such as hallucinations and prompt injection. CTOs need standards and comparable measurements. In specialized work such as financial analysis, the evaluation problem requires domain experts who can validate whether an agent's work is correct. John says that expert review may be expensive, but it is justified when an incorrect result can cause financial loss or cost someone their job.

### Fast, application-specific retrieval evals enable cheap experiments
[3:37:07](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=13027s)
Jeff Huber recommends fast evals for retrieval systems. A fast eval is a set of query and document pairs where each query is associated with a document that should be retrieved. Running the queries and checking whether the expected documents appear is inexpensive and quick, so teams can compare embedding models without waiting hours or relying only on public benchmarks. Synthetic queries can help when there are no users, but they need to be made representative of real queries rather than copied from overly clean documents. Jeff describes a comparison in which Voyage 3 large performed best for a particular application even though other models had stronger public benchmark results. The data from the application determined the choice.

### Conversation clustering turns eval results into product decisions
[3:44:21](https://www.youtube.com/watch?v=Vqsfn9rWXR8&t=13461s)
Jason Liu argues that a single aggregate score does not explain what to build next. Once a product has too many conversations for manual review, teams can extract structured metadata such as summaries, tools used, errors, frustration, and satisfaction. They can then embed and cluster conversations into groups such as authentication errors, SEO requests, or data visualization. Comparing performance across those groups reveals where the product is weak and where usage is concentrated. A low score on its own is hard to act on. A low score for a heavily used category can justify building a tool, changing a workflow, or adding a data source. The resulting analysis can guide the roadmap and identify work to fix, build, or ignore.

## Notable quotes
- "A prompt is a horrible abstraction for programming and this needs to be fixed ASAP." (21:50)
- "The ultimate judge are your users." (46:04)
- "There really is no shortcut here. You really have to do the work and understand what your court looks like." (58:50)
- "Improvement without measurement is limited and imprecise." (1:04:04)
- "Start small, look for structure, understand that structure, and start comparing your KPIs." (3:52:55)

## Tools & references mentioned
- Bitter Lesson
- Rich Sutton
- DSPy
- BERT
- DaVinci 2
- GPT-4o4-mini
- Zapier Agents
- Braintrust
- Vercel
- V0
- AI SDK
- Bolt Foundry
- OpenRouter
- GPT-4.1
- Claude 3.7 Sonnet
- Claude 4 Sonnet
- Gemini 2.5 Flash
- Gemini 2.5 Pro
- Chroma
- Krea
- Mozilla AI
- Arthur AI
- Clean-FID
- FID
- JPEG
- MP3
- MP4
- MTEB
- Voyage 3 large
- Weights & Biases
- Weave
- Cura
- Anthropic
- OpenAI

## Who should watch
- You are building an agent and need a practical way to turn production failures, tool calls, and user reactions into eval cases.
- Your retrieval or generative-media system looks good on public benchmarks, but you do not know whether it works for your own users and data.
- You need to compare models, prompts, or workflows before launch and connect offline scores with production behavior and business outcomes.

## Related talks

- [Evals 101](https://aietalks.com/talks/evals-101) (Doug Guthrie, Braintrust, 48:31)
- [Evals Are Not Unit Tests](https://aietalks.com/talks/evals-are-not-unit-tests) (Ido Pesok, Vercel, 15:22)
- [Five hard earned lessons about Evals](https://aietalks.com/talks/five-hard-earned-lessons-about-evals) (Ankur Goyal, Braintrust, 19:46)
- [Why should anyone care about Evals?](https://aietalks.com/talks/why-should-anyone-care-about-evals) (Manu Goyal, Braintrust, 05:41)
- [How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity](https://aietalks.com/talks/how-to-run-evals-at-scale-thinking-beyond-accuracy-or-similarity) (Muktesh Mishra, Adobe, 09:25)