# Evals Are Not Unit Tests

Ido Pesok, Vercel | AI Engineer World's Fair 2025 | 15:22

Source: https://www.youtube.com/watch?v=L8OoYeDI_ls
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/evals-are-not-unit-tests
Published: 2025-08-06
Tags: evals, reliability, testing

## TL;DR
- Application evals test whether an AI system works across the queries users actually make, including difficult cases that ordinary unit tests miss.
- A useful eval dataset maps the domain of user queries, keeps the data fixed, and varies prompts, retrieval, preprocessing, or models in the task.
- Simple deterministic scores are easier to debug and share, while human review is acceptable when code cannot capture the right judgment.

## Summary
Ido Pesok explains how to evaluate AI applications after they leave the model lab. He uses Vercel v0's fictional Fruit Letter Counter to show how an app can work repeatedly in a demo and still fail when users ask a slightly different question. Unit tests can cover the surrounding software while missing the unreliable model behavior that matters most. Pesok compares an eval to basketball practice: the user queries define the court, each result is a made or missed shot, and the dataset should cover the whole area where users might play. He recommends collecting real queries from feedback, logs, forums, and social media. Keep those queries fixed while changing prompts, retrieval, preprocessing, or models. Scores should be simple and preferably deterministic. Running evals in CI lets a team see whether a change turns failures into successes or creates regressions elsewhere.

## Key ideas
### AI applications can pass demos and still fail in production
[00:57](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=57s)
Pesok says the model is now out of the research lab and must work for a specific app, its users, and its data. He illustrates the problem with Fruit Letter Counter, a small app that counts letters in fruit. GPT-4.1 answered that there were three Rs in strawberry, and the answer was repeated successfully twice. After launch, a user reported that the app answered two. A more elaborate prompt worked ten times in a row, then failed on a request containing strawberry, banana, pineapple, mango, kiwi, dragon fruit, apple, and raspberry. Pesok says this pattern applies to large AI products too: an app can look good in a demo and expose hallucinations after deployment.

### The difficult part of an AI app can escape ordinary software tests
[00:35](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=35s)
Pesok argues that an AI app may have unit tests for every function and end-to-end tests for authentication, login, and sign-out, while the model-dependent part still fails. In his example, 95% of the app works 100% of the time, but the remaining 5% can break the user experience. Prompt engineering and chain-of-thought instructions may improve a few tested examples without covering the requests users invent in production. The evaluation problem therefore concerns behavior across a domain of possible queries, rather than a single expected output for one fixed input.

### An eval dataset should describe the whole court where users play
[03:33](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=213s)
Pesok compares evaluation to basketball practice. The basket is the desired result, a blue point is a made shot, and a red point is a miss. Shots farther from the basket are harder, and points outside the court do not count. For Fruit Letter Counter, counting the Rs in strawberry is near the basket, while counting Rs across a large array of fruits is farther away. Replacing every vowel with an R is an extreme query that still belongs to the app's domain. Counting syllables in carrot is out of bounds. The dataset should include the boundaries and the difficult regions of the real user domain.

### Real user data is the foundation of useful evals
[07:53](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=473s)
Pesok recommends collecting thumbs-up and thumbs-down feedback, while acknowledging that it can be noisy. If an app has observability, a team can inspect random samples from its logs, such as a hundred samples once a week, to learn how people use the product. Community forums often contain reports of model problems. X and Twitter can also provide examples, although they can be noisy. He is direct that there is no shortcut: the team must do the work of understanding its users' queries. Once the data is mapped, the red areas show where future engineering effort should go.

### Keep user data constant while changing the evaluation task
[09:13](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=553s)
Pesok recommends putting constants in the data and variables in the task. A query such as 'How many Rs are in strawberry?' stays in the dataset. The task can then try different system prompts, preprocessing steps, retrieval-augmented generation, or models against that same query set. This avoids rebuilding the dataset whenever the system prompt changes and makes comparisons clearer. He mentions Braintrust as supporting this structure, and he recommends using AI SDK middleware to share preprocessing, retrieval, and system-prompt logic between the production API route and the evals. The evaluation should resemble the code that runs in the real application.

### Simple scores make failures easier to debug
[10:45](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=645s)
Scoring depends on the application. For Fruit Letter Counter, the score can check whether the output contains the correct number. Pesok recommends deterministic scoring and pass-fail results when possible because debugging produces many inputs and logs. Overly complicated scoring can make the eval hard to understand and share across teams. The team should first ask what it is looking for when an example fails, then write a check for that condition. Some writing tasks are too difficult to score in code, so human review is acceptable. He also suggests adding a small instruction such as outputting the answer inside tags, which makes string matching easier without requiring that format in production.

### CI eval reports expose regressions across the query domain
[12:35](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=755s)
Pesok recommends adding evals to CI. When a colleague changes a prompt in a pull request, the eval should run the task across the dataset and produce a report of improvements and regressions. The basketball view makes the comparison concrete: the team can see whether the change turned red tiles blue, or fixed one area while breaking another. The same practice can compare models, retrieval changes, and system-prompt changes. This gives a team evidence for whether a change helped the app instead of relying on a few successful manual tests.

### Regular practice gives teams a way to improve reliability
[13:40](https://www.youtube.com/watch?v=L8OoYeDI_ls&t=820s)
Pesok describes evals as a practice environment for the application. Running them on a regular schedule gives the team a view of where the system fails and whether a change caused a regression. In the question period, he compares this with a basketball player who scores well overall but misses more often in particular areas. Repeated runs can also show how often the same question succeeds, such as whether it works four out of five times. He connects this measurement to better reliability and quality, higher conversion and retention, and less support and operations work.

## Notable quotes
- "By nature, LLMs can be very unreliable." (02:42)
- "To make good evals, you must understand your court." (07:12)
- "You want to put constants in data, variables in the task." (09:20)
- "Improvement without measurement is limited and imprecise." (13:27)
- "There really is no shortcut here. You really have to do the work and understand what your court looks like." (08:21)

## Tools & references mentioned
- Vercel
- v0
- GitHub
- AI SDK
- GPT-4.1
- Braintrust
- ChatGPT
- NBA Finals

## Who should watch
- You are shipping an AI feature that looks reliable in demos but fails on requests you did not anticipate.
- Your team has unit and end-to-end tests, yet lacks a dataset of real user queries for model behavior.
- You need a practical way to compare prompt, retrieval, preprocessing, or model changes in CI.

## Editor's note

From the pack [Evals that survive production](https://aietalks.com/packs/production-evals):

Zapier records tool interactions so a failed run can become a test. Hetzel calls the next step rerunning production, while Feyzkhanov rebuilds the environment around a captured trace and Gupta evaluates the whole workflow. Kitaru records an agent run and replays that case after a model, prompt, tool or harness changes, so the team can compare the route as well as the final result.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Evals](https://aietalks.com/talks/evals) (Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI, 3:59:12)
- [Evals 101](https://aietalks.com/talks/evals-101) (Doug Guthrie, Braintrust, 48:31)
- [How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity](https://aietalks.com/talks/how-to-run-evals-at-scale-thinking-beyond-accuracy-or-similarity) (Muktesh Mishra, Adobe, 09:25)
- [Why should anyone care about Evals?](https://aietalks.com/talks/why-should-anyone-care-about-evals) (Manu Goyal, Braintrust, 05:41)
- [Practical Tactics to Build Reliable AI Apps](https://aietalks.com/talks/practical-tactics-to-build-reliable-ai-apps) (Dmitry Kuchin, Multinear, 14:55)