Evals give teams a repeatable way to tell whether a model, prompt, or application change improved or damaged an AI system.
2
A useful eval needs a task, a data set of examples, and at least one score, which can be code-based or judged by an LLM.
3
Production traces, user feedback, and human reviews can be turned into new data set examples for offline testing.
Summary
Doug Guthrie explains how to build an evaluation process around an AI application, starting with prompt experiments and continuing into production monitoring. He defines an eval as a structured test of quality, reliability, or correctness. The basic ingredients are a task, a data set, and one or more scores. Scores can use code or an LLM judge, and Braintrust provides both a platform workflow and SDKs for Python, TypeScript, Go, Java, and Kotlin. Guthrie recommends creating a small baseline instead of waiting for a perfect test set. Teams can compare prompts and models in experiments, run evals in CI/CD, trace production requests, apply sampled online scores, and alert on score drops. Human review and user feedback then provide examples that can be added to offline data sets. The workshop uses a GitHub change-log application to demonstrate the workflow.
Guthrie says evals answer practical questions such as whether changing the underlying model or prompt made an application better or worse. Large language models produce non-deterministic outputs, so teams need a rigorous process rather than relying on impressions. Evals can detect regressions in code and give development more structure. He relays Braintrust CEO Ankur Goyle's framing that evals let teams play offense: they help create and improve applications instead of only defending against failures with tests. Offline and online evals also connect development work with what users experience in production.
A small baseline is more useful than waiting for a perfect test set
Guthrie advises teams to start by creating a baseline and then improve it. They do not need to wait until they have a carefully curated golden data set. He gives a diagnostic matrix: good output with a low score suggests improving the eval, while bad output with a high score suggests improving the eval or its scoring. The point is to establish a foundation that gives the team a direction for further work. This approach lets developers begin measuring an application before they have settled on every test case or scoring rule.
The task is the code or prompt being evaluated. It can be a single prompt or a larger agentic workflow that calls tools, as long as it has an input and an output. The data set contains real-world examples against which the task runs. A score supplies the evaluation logic. An LLM judge can assess an output against criteria such as excellent, fair, or poor, while a code-based score can apply a heuristic or binary condition. Data set records require an input and can also include an expected output, metadata, and other fields.
Focused scores make evaluation results easier to use
Braintrust supports scores written in TypeScript or Python, either in the platform or in a team's codebase. Its LLM-as-a-judge scores use criteria supplied by the user, and the auto evals package provides ready-made code-based and LLM-based scores. Guthrie recommends using a stronger model for scoring when the application itself uses a cheaper model. He also advises splitting broad judgments into focused scores. For a change-log generator, separate scores could assess accuracy, formatting, and correctness. Score prompts should be tested in the playground and should contain only relevant context.
Playgrounds and experiments support fast offline iteration
In the Braintrust playground, a developer can load prompts or agents, choose an underlying model, attach a data set, add scores, and run the task across the examples. Guthrie demonstrates comparing a base prompt with a version using GPT-4.1 for a GitHub change-log application. The summary view shows score differences for completeness, accuracy, and formatting. Experiments preserve snapshots over time, so teams can see whether later prompt or model changes improved the application. He also demonstrates a beta loop feature that can use evaluation results to propose a prompt change and run the evaluation again.
The platform and SDK support different team workflows
Teams can define prompts, data sets, and scores in code, push them into Braintrust, and keep those assets under version control. They can also define the eval itself in code with the SDK by supplying a data set, task, and scores, then run it as an experiment in the platform. Braintrust offers Python and TypeScript SDKs, along with Go, Java, and Kotlin options. Guthrie says the same evals can run from CI/CD, where a team can check whether configured scores improved or worsened. This lets organizations choose between platform-based iteration and code-first workflows.
Production traces connect online monitoring to offline testing
To monitor live traffic, a team initializes a logger, points it at a Braintrust project, and instruments the application. Wrapping an OpenAI client captures metrics such as tokens, duration, and cost. Trace decorators and lower-level span APIs can record tool calls, inputs, outputs, and metadata. Guthrie stresses that consistent span structure matters because it makes production records easier to review and add to existing data sets. Online scoring can run selected scores against sampled traffic rather than every request, and automation can alert the team when a score falls below a chosen threshold.
Human review and user feedback produce new test cases
Braintrust lets teams create filtered views of production logs and open a reduced human review interface. Reviewers can apply configured scores or add more free-form feedback. Applications can also send user feedback such as thumbs up, thumbs down, and comments. Guthrie shows how a team could filter for records where user feedback equals zero, save that filter as a view, and investigate the failures. Selected production rows can then be added to the data set used by offline evals. He recommends giving human reviewers a rubric so different people apply similar scoring criteria.
"This isn't like hey I think this got better that output looks better. There is actual rigor behind this now."23:23
Who should watch
You are building an LLM or agent application and need a starting point for testing prompts, models, and tool-using workflows.
Your team has production traces or user feedback but has not connected them to the examples used in pre-production testing.
You need to run evals from code or CI/CD, compare multiple production models, or give non-engineers a way to review outputs.
