# Evals that survive production

A pack of 10 talks from the AI Engineer YouTube channel, in the order to watch them. 3h 31m of video.
Page: https://aietalks.com/packs/production-evals

An eval set assembled before launch can tell you whether a change helped on the cases you expected. It cannot tell you what frustrated users will try next, whether a judge agrees with the people doing the work, or what happened after an agent called a tool. A production eval loop starts by defining the actual range of user requests and locating failures inside the application. Traces and human explanations become regression cases; calibrated judges and simulations make more of them repeatable. Evaluation then stays connected to the running system, where changes to users, tools and agent behavior keep supplying new evidence. The suite survives only if the team continues to collect failures, test its graders and revise what good means after launch.

## This pack is for you if

- Your AI feature passes demos and ordinary tests, but users keep finding requests the team never considered.
- You have traces, feedback and expert review, but they do not yet become repeatable regression cases or trustworthy graders.
- Your offline score improves while production outcomes, tool failures or escalations stay flat.

## The talks, in order

### 1. Evals Are Not Unit Tests

Ido Pesok, Vercel | 15:22 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=L8OoYeDI_ls
Summary: https://aietalks.com/talks/evals-are-not-unit-tests.md

Why first: Pesok establishes what an application eval has to cover: not one expected answer, but the range of requests real users can make. Dhinakaran follows those requests into the application and shows why one overall score cannot locate a bad router decision.

### 2. Lessons from the Trenches: Building LLM Evals That Work IRL

Aparna Dhinakaran, Arize AI | 18:49 | AI Engineer World's Fair 2024
Video: https://www.youtube.com/watch?v=nbZzSC5A6hs
Summary: https://aietalks.com/talks/lessons-from-the-trenches-building-llm-evals-that-work-irl.md

Why second: Pesok maps the requests an application must survive. Dhinakaran maps the system they travel through, from the router down to spans and components, so an engineer can tell what failed. Goyal then asks how that evaluation work stays useful as the model and application change.

### 3. Five hard earned lessons about Evals

Ankur Goyal, Braintrust | 19:46 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=a4BV0gGmXgA
Summary: https://aietalks.com/talks/five-hard-earned-lessons-about-evals.md

Why here: Dhinakaran shows where to attach evals. Goyal supplies the maintenance discipline around them: keep the data current, treat scoring as the product specification, and make model comparisons cheap. Zapier shows where the next cases come from once users get hold of the agent.

### 4. Turning Fails into Features: Zapier's Hard-Won Eval Lessons

Rafal Willinski & Vitor Balocco, Zapier | 16:15 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=blrovBxxN9o
Summary: https://aietalks.com/talks/turning-fails-into-features-zapiers-hard-won-eval-lessons.md

Why here: Goyal describes the evaluation machinery. Willinski and Balocco feed it with failed runs, tool traces and signs of user frustration. They also find that narrow unit evals can punish a better model for taking a different route. Hetzel follows with a practical path from those observations to tested graders.

### 5. The maturity phases of running evals

Phil Hetzel, Braintrust | 18:34 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=FB-MLPhL9Ms
Summary: https://aietalks.com/talks/the-maturity-phases-of-running-evals.md

Why here: Zapier has a flywheel collecting failures. Hetzel explains how a team grows into it: preserve why reviewers passed or failed a run, derive narrow checks, and recreate the external state behind tool calls. Lyft then tests the synthetic users and judges that decide whether an agent is ready to launch.

### 6. Build Evals That Actually Matter

Nick Ung & Akshay Sharma, Lyft | 37:45 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=3z2uT5aDx_Y
Summary: https://aietalks.com/talks/build-evals-that-actually-matter.md

Why here: Hetzel turns expert explanations into graders. Ung and Sharma show how both sides can lie: a helpful simulator makes the agent look better than it is, while an uncalibrated judge produces a score nobody should ship on. Mabrouk makes that judge the subject of the next experiment.

### 7. Judge the Judge: Building LLM Evaluators That Actually Work with GEPA

Mahmoud Mabrouk, Agenta AI | 40:51 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=X4dEHRzBLmc
Summary: https://aietalks.com/talks/judge-the-judge-building-llm-evaluators-that-actually-work-with-gepa.md

Why here: Lyft validates a binary judge against held-out human labels. Mabrouk pushes further, showing that prompt optimization can improve measured agreement without making the evaluator trustworthy. Feyzkhanov widens the target from one judge to the complete environment in which an agent run can be repeated.

### 8. From Agent Traces to Agent Simulations

Rustem Feyzkhanov, Snorkel AI | 20:24 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=Ib5t2RLtxvM
Summary: https://aietalks.com/talks/from-agent-traces-to-agent-simulations.md

Why here: Mabrouk makes one evaluator auditable. Feyzkhanov makes the full run repeatable with a task, an environment, an oracle solution and layered verification. That lets a company test the private system it actually ships. Gupta follows by moving the same workflow-level evidence into production operations.

### 9. Production Evals For Agentic AI Systems

Nishant Gupta, Meta Superintelligence Labs | 08:12 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=vljxQZfJ9wY
Summary: https://aietalks.com/talks/production-evals-for-agentic-ai-systems.md

Why here: Feyzkhanov reconstructs a production case offline. Gupta closes the loop online, measuring task outcomes, tool behavior, escalation and recovery as the agent runs. Koc ends with the problem that remains: users and agents keep changing faster than a fixed suite can follow.

### 10. Malleable Evals: Why Are We Evaluating Adaptive Systems with Static Tests?

Vincent Koc, OpenClaw | 15:05 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=4VhbYlfC7Gs
Summary: https://aietalks.com/talks/malleable-evals-why-are-we-evaluating-adaptive-systems-with-static-tests.md

Why last: Gupta puts continuous evaluation around a production agent. Koc asks what happens when that agent, its tools and its users change. His answer is to draw new cases from traces while judging the outcome instead of prescribing one acceptable route. The pack ends with an eval suite that has to change because the product does.

## Editor's note

Zapier records tool interactions so a failed run can become a test. Hetzel calls the next step rerunning production, while Feyzkhanov rebuilds the environment around a captured trace and Gupta evaluates the whole workflow. Kitaru records an agent run and replays that case after a model, prompt, tool or harness changes, so the team can compare the route as well as the final result.

Written by the AIE Talks editors (the Kitaru team), not by any of the speakers.