# Why Building an Eval Platform Is Harder Than It Looks

Hossein Niazmandi, Braintrust | AI Engineer World's Fair 2026 | 17:47

Source: https://www.youtube.com/watch?v=mUQoVz7THu0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/why-building-an-eval-platform-is-harder-than-it-looks
Published: 2026-10-06
Tags: evals, observability, tracing

## TL;DR
- Agent quality requires both offline evals before launch and observability after deployment.
- Eval platforms grow from spreadsheets into systems that support datasets, scoring, collaboration, production traces, and long-running analysis.
- Coding agents can run evals and suggest changes, while humans review the results and decide what to ship.

## Summary
Hossein Niazmandi explains why agent quality platforms involve much more than a spreadsheet or a nicer interface. Evals test agents before production, while observability tracks real user interactions after launch. Both are needed because LLM behavior is variable and production failures can differ from the cases teams anticipated offline. He describes a progression from spreadsheets, to custom UIs, to side-by-side experiments for PMs and domain experts, and then to a flywheel that turns production failures into new test cases. The technical problem grows when traces become large, nested, semi-structured JSON that must support both real-time investigation and long-running analysis. Braintrust built BTQL for this workload. Niazmandi also describes coding agents that find traces, run evals, and propose iterations. Humans still review the outcomes and choose what to release. The talk is candid that the hard work includes data, permissions, masking, query systems, and workflows for different roles.

## Key ideas
### Agent quality has an offline pillar and a production pillar
[00:51](https://www.youtube.com/watch?v=mUQoVz7THu0&t=51s)
Braintrust frames agent quality around evals and observability. Evals happen before an agent reaches production, when teams test behavior and build confidence before shipping. Observability begins when the agent interacts with real users, allowing teams to monitor whether the assumptions from offline development hold up in practice. The two activities address the same goal at different points in the lifecycle. Evals reduce uncertainty before launch, while observability shows what happens after launch and provides evidence for further improvement.

### LLM variability makes confidence an engineering problem
[02:08](https://www.youtube.com/watch?v=mUQoVz7THu0&t=128s)
LLMs can reason across different domains and handle many kinds of requests, but their behavior is highly variable. That flexibility creates risk when an agent becomes part of a customer experience. Niazmandi names inconsistent behavior as a brand risk, wrong answers or actions as a compliance risk, and difficult debugging as a cost and maintenance risk. Evals give teams a way to test expected behavior before launch instead of relying on a single successful demonstration.

### A spreadsheet is a valid start, but it cannot support the full workflow
[04:01](https://www.youtube.com/watch?v=mUQoVz7THu0&t=241s)
A basic eval system needs three things: a way to execute an agent against test inputs, a way to view outputs or scores, and a set of inputs that can invoke the agent. A spreadsheet can cover this first version and has a low barrier to entry. Its limits appear quickly. Analytics become manual, human scoring is difficult to manage, collaboration is weak, and domain experts are often left out. A custom UI improves access for people such as PMs, but it can remain a reporting tool without durable data and long-horizon analysis.

### Eval maturity moves toward experiments for more roles
[09:50](https://www.youtube.com/watch?v=mUQoVz7THu0&t=590s)
The next stage lets non-technical users change the system prompt, switch the underlying model, adjust parameters, and compare results side by side. This brings PMs and subject-matter experts into the evaluation process. The test cases are still usually curated offline, though, so teams can ship an agent with a good idea of expected behavior and still be blind to the steps and failures that occur in production.

### The improvement loop turns production failures into test cases
[06:46](https://www.youtube.com/watch?v=mUQoVz7THu0&t=406s)
The target workflow starts with a production application, observes failure modes across defined dimensions, and captures those failures as offline test cases. Teams then iterate against them, improve the agent, and check for regressions before shipping again. The loop continues because user behavior and agent implementations change over time. A complete platform needs to record inputs, outputs, and every step of an agent trace, then connect those production observations to offline evaluation.

### Agent traces create a systems problem
[11:49](https://www.youtube.com/watch?v=mUQoVz7THu0&t=709s)
Production traces can be semi-structured JSON with deeply nested content. Niazmandi says some teams log a movie's worth of data in one interaction, with payloads reaching hundreds of megabytes. That creates different requirements from traditional observability data. Teams need real-time queries as data arrives, long-running queries for tuning and annotation, large-volume aggregation, and snapshots of data at a point in time. Typical cloud data warehouses can struggle with this combination of size, structure, and read patterns.

### Coding agents can run evaluations while people judge the outcome
[14:28](https://www.youtube.com/watch?v=mUQoVz7THu0&t=868s)
Eval platforms increasingly need to support agents as users. A coding agent can receive a natural-language instruction such as finding poor user experiences in the last 24 hours and running the relevant evals. Because it can access the codebase and underlying infrastructure, it can execute the workflow and log results. Braintrust also describes using inference over tracing data to surface failures humans did not define in advance, such as silent failures or repeated prompts caused by user frustration. Coding agents can suggest changes, but humans decide which iteration is suitable for production.

## Notable quotes
- "There are two key pillars of agent quality. The first is evals. The second pillar is observability." (00:51)
- "Spreadsheet-based evals are not wrong. However, they're often the first step for a useful version of eval workflow." (04:48)
- "Agent traces are nasty. They are semi-structured JSON." (11:58)
- "Building an eval platform truly becomes a systems problem." (13:33)
- "Ultimately, the humans responsibility is to review the outcome." (16:54)

## Tools & references mentioned
- Braintrust
- BTQL
- Salesforce
- Databricks
- LLMs

## Who should watch
- You are moving from ad hoc evals toward a shared workflow for engineers, PMs, and domain experts.
- Your production traces are large or nested, and you need both immediate failure investigation and historical analysis.
- You want coding agents to run evaluations or find failure cases, while keeping human review in the release decision.

## Editor's note

Hossein Niazmandi explains that teams can ship an agent with a good idea of expected behavior while remaining blind to failures that occur in production. Kitaru records each model call and tool result from a real agent run, then replays the run with the same inputs and responses. Teams can inspect a failed run after changing the model, prompt, or code.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Why building eval platforms is hard](https://aietalks.com/talks/why-building-eval-platforms-is-hard) (Phil Hetzel, Braintrust, 25:39)
- [Build Evals That Actually Matter](https://aietalks.com/talks/build-evals-that-actually-matter) (Nick Ung & Akshay Sharma, Lyft, 37:45)
- [The maturity phases of running evals](https://aietalks.com/talks/the-maturity-phases-of-running-evals) (Phil Hetzel, Braintrust, 18:34)
- [Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work](https://aietalks.com/talks/engineering-better-evals-scalable-llm-evaluation-pipelines-that-work) (Dat Ngo & Aman Khan, Arize AI, 24:46)
- [Turning Fails into Features: Zapier's Hard-Won Eval Lessons](https://aietalks.com/talks/turning-fails-into-features-zapiers-hard-won-eval-lessons) (Rafal Wilinski & Vitor Balocco, Zapier, 16:15)