# Mastering AI Evaluation: From Playground to Production Doug & Carlos Essan, Braintrust | AI Engineer World's Fair 2025 | 1:25:08 Source: https://www.youtube.com/watch?v=9iN-cPnp7xg Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/mastering-ai-evaluation-from-playground-to-production Published: 2025-07-01 Tags: evals, human-in-the-loop, observability, testing ## TL;DR - AI evaluations give teams empirical evidence for choosing models, changing prompts, catching regressions, and checking quality before production. - A useful evaluation combines a task, a data set, and scores, with synthetic examples as a starting point and real user logs added over time. - Production logging, online scoring, and human review create a feedback loop that turns weak outputs into better data sets and offline tests. ## Summary Doug and Carlos Essan present Braintrust through a hands-on evaluation workflow. They define an evaluation as a structured test made from a task, a data set, and one or more scores. The task can be a prompt, a multi-turn conversation, a tool-using workflow, or an agent. Teams can start with synthetic data, then ground the data set in real user traffic and human review. The workshop compares the Braintrust playground for quick prompt and model iteration with experiments for historical comparison, and shows how the same work can be versioned and run through the SDK. For production, they demonstrate logging prompts, outputs, tool calls, token counts, latency, and errors, then applying online scoring to sampled traffic. Human reviewers can inspect low-scoring traces, add feedback, and promote useful examples into data sets. The speakers are candid that LLM judges vary, so teams should validate them against human judgments and deterministic scores. ## Key ideas ### Evaluations give development teams evidence for AI changes [03:47](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=227s) Doug says evaluations answer practical questions about model choice, cost, edge-case performance, brand voice, regressions, bugs, and troubleshooting. Strong language-model performance alone does not guarantee consistent results. A prompt change that looks better in one example can make the application worse elsewhere. Teams therefore need an empirical testing framework before pushing changes into production. Doug argues that evaluations can reduce manual review, speed releases, help optimize model cost, and let technical and non-technical people take part in prompt and model decisions. ### Every evaluation has a task, a data set, and a score [17:36](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=1056s) The task is the code or prompt being tested, with an input and an output. It can be a single language-model call, a full agentic workflow, a multi-turn conversation, or a workflow that uses tools and retrieval. The data set contains real-world examples or test cases, with an input required and expected output and metadata available as optional fields. The score grades the output and must produce a value from zero to one. Scores can use an LLM as a judge, code-based logic, or both. ### Synthetic examples are a reasonable start, but real traffic improves the data set [08:40](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=520s) The speakers recommend using synthetic data to get an initial evaluation running quickly. As the system matures, teams should add examples from logs of real user interactions. The data set does not need to cover every use case before work begins. Doug recommends starting small and iterating, even when there are only a few rows. Human review can then establish ground truth, improve expected outputs, and expand the test cases toward the situations users actually encounter. ### The score matrix tells you whether to fix the evaluator or the application [10:36](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=636s) Doug describes four cases based on human judgment and the evaluation score. A good output with a high score means the evaluator agrees with the reviewer. A good output with a low score suggests that the evaluator does not represent human quality and needs work. A bad output with a high score points to the same evaluator problem. A bad output with a low score means the evaluation is working and the application needs improvement. The matrix helps teams decide what to investigate instead of treating every low score as an application defect. ### LLM judges need narrow criteria and human validation [23:44](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=1424s) LLM-as-judge scores work well for subjective or contextual qualities, while code-based scores fit exact or binary conditions. The speakers recommend combining both. A judge should focus on one criterion instead of handling several at once, and a higher-quality model can grade a cheaper production model. Teams should also evaluate the judge itself by comparing its decisions with human reviews. Reading the judge's rationale in the logs helps tune its prompt. Absolute scores matter less than changes against a previous baseline. ### The playground supports quick comparisons, while experiments preserve history [17:08](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=1028s) Braintrust's playground is intended for quick prompt, model, data set, and score iteration, including A/B comparisons. A run evaluates the selected tasks against the selected data set and scores in parallel. The result lets a team compare prompt or model changes using actual evaluation data rather than a single subjective inspection. Saving the work as an experiment creates a longer-lived record for comparing results over time. The speakers describe the playground as more ephemeral and experiments as historical, although the two areas have become increasingly similar. ### The SDK brings evaluation definitions into version control and CI [38:03](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=2283s) The Braintrust SDK supports TypeScript and Python workflows in which prompts, scores, data sets, and evaluation definitions live in the application repository. A push command sends these resources to the Braintrust project, while an eval command runs evaluations and records them in the experiments view. This lets teams change a prompt or model alongside application code and compare the resulting scores. The speakers say evaluations can dynamically call the changing application, so a pull request can run tests against the updated task instead of a permanently copied version. ### Production traces turn online failures into offline test cases [55:35](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=3335s) The production workflow starts with logging. Wrapping an LLM client can capture prompts, responses, token counts, latency, and errors, while tracing functions and adding metadata provides more detail. Online scoring evaluates selected live traffic, with sampling controlled by the team. Rules can target the root span or a nested span, and teams can create saved views for low scores or negative user feedback. Reviewers can inspect those traces and add them to a data set, closing the loop between production monitoring and offline evaluation. ### Human review supplies ground truth for both applications and judges [1:04:50](https://www.youtube.com/watch?v=9iN-cPnp7xg&t=3890s) Subject-matter experts, product managers, or end users can review traces manually, label them, score them, and add comments. Applications can also collect direct feedback such as thumbs-up and thumbs-down ratings. Teams can filter those signals in Braintrust, inspect negative examples, and add selected traces to a data set. Human labels also help test whether an LLM judge matches the quality standard people expect. The speakers describe this as necessary because LLM outputs are non-deterministic and automated scores can miss important nuance. ## Notable quotes - "The best LLMs don't always guarantee consistent performance." (04:44) - "The important piece is to try to use both." (15:47) - "You really just want to get started quickly." (1:16:38) - "This isn't vibe check." (34:19) - "You don't need to run an eval as the user's interacting with it." (1:06:04) ## Tools & references mentioned - Braintrust - OpenAI - GPT-4.1 - Amazon Bedrock - Vercel AI SDK - OpenTelemetry - Terraform - Vault - GitHub - LangSmith ## Who should watch - You are changing prompts or models and need evidence that an apparent improvement has not caused regressions in other cases. - Your team has production traces or user feedback but no practical process for turning them into evaluation data. - You need to combine automated scores with reviewers who understand the domain and can check whether an LLM judge is behaving sensibly. ## Related talks - [Evals 101](https://aietalks.com/talks/evals-101) (Doug Guthrie, Braintrust, 48:31) - [How to build world-class AI products](https://aietalks.com/talks/how-to-build-world-class-ai-products) (Sarah Sachs, Notion & Carlos Esteban, Braintrust, 1:43:46) - [Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)](https://aietalks.com/talks/strategies-for-llm-evals-guidellm-lm-eval-harness-openai-evals-workshop) (Taylor Jordan Smith, Red Hat, 32:28) - [Shipping AI That Works: An Evaluation Framework for PMs](https://aietalks.com/talks/shipping-ai-that-works-an-evaluation-framework-for-pms) (Aman Khan, Arize, 1:26:16) - [Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work](https://aietalks.com/talks/engineering-better-evals-scalable-llm-evaluation-pipelines-that-work) (Dat Ngo & Aman Khan, Arize, 24:46)