Agent quality requires both offline evals before launch and observability after deployment.
2
Eval platforms grow from spreadsheets into systems that support datasets, scoring, collaboration, production traces, and long-running analysis.
3
Coding agents can run evals and suggest changes, while humans review the results and decide what to ship.
Summary
Hossein Niazmandi explains why agent quality platforms involve much more than a spreadsheet or a nicer interface. Evals test agents before production, while observability tracks real user interactions after launch. Both are needed because LLM behavior is variable and production failures can differ from the cases teams anticipated offline. He describes a progression from spreadsheets, to custom UIs, to side-by-side experiments for PMs and domain experts, and then to a flywheel that turns production failures into new test cases. The technical problem grows when traces become large, nested, semi-structured JSON that must support both real-time investigation and long-running analysis. Braintrust built BTQL for this workload. Niazmandi also describes coding agents that find traces, run evals, and propose iterations. Humans still review the outcomes and choose what to release. The talk is candid that the hard work includes data, permissions, masking, query systems, and workflows for different roles.
Agent quality has an offline pillar and a production pillar
Braintrust frames agent quality around evals and observability. Evals happen before an agent reaches production, when teams test behavior and build confidence before shipping. Observability begins when the agent interacts with real users, allowing teams to monitor whether the assumptions from offline development hold up in practice. The two activities address the same goal at different points in the lifecycle. Evals reduce uncertainty before launch, while observability shows what happens after launch and provides evidence for further improvement.
LLM variability makes confidence an engineering problem
LLMs can reason across different domains and handle many kinds of requests, but their behavior is highly variable. That flexibility creates risk when an agent becomes part of a customer experience. Niazmandi names inconsistent behavior as a brand risk, wrong answers or actions as a compliance risk, and difficult debugging as a cost and maintenance risk. Evals give teams a way to test expected behavior before launch instead of relying on a single successful demonstration.
A spreadsheet is a valid start, but it cannot support the full workflow
A basic eval system needs three things: a way to execute an agent against test inputs, a way to view outputs or scores, and a set of inputs that can invoke the agent. A spreadsheet can cover this first version and has a low barrier to entry. Its limits appear quickly. Analytics become manual, human scoring is difficult to manage, collaboration is weak, and domain experts are often left out. A custom UI improves access for people such as PMs, but it can remain a reporting tool without durable data and long-horizon analysis.
Eval maturity moves toward experiments for more roles
The next stage lets non-technical users change the system prompt, switch the underlying model, adjust parameters, and compare results side by side. This brings PMs and subject-matter experts into the evaluation process. The test cases are still usually curated offline, though, so teams can ship an agent with a good idea of expected behavior and still be blind to the steps and failures that occur in production.
The improvement loop turns production failures into test cases
The target workflow starts with a production application, observes failure modes across defined dimensions, and captures those failures as offline test cases. Teams then iterate against them, improve the agent, and check for regressions before shipping again. The loop continues because user behavior and agent implementations change over time. A complete platform needs to record inputs, outputs, and every step of an agent trace, then connect those production observations to offline evaluation.
Production traces can be semi-structured JSON with deeply nested content. Niazmandi says some teams log a movie's worth of data in one interaction, with payloads reaching hundreds of megabytes. That creates different requirements from traditional observability data. Teams need real-time queries as data arrives, long-running queries for tuning and annotation, large-volume aggregation, and snapshots of data at a point in time. Typical cloud data warehouses can struggle with this combination of size, structure, and read patterns.
Coding agents can run evaluations while people judge the outcome
Eval platforms increasingly need to support agents as users. A coding agent can receive a natural-language instruction such as finding poor user experiences in the last 24 hours and running the relevant evals. Because it can access the codebase and underlying infrastructure, it can execute the workflow and log results. Braintrust also describes using inference over tracing data to surface failures humans did not define in advance, such as silent failures or repeated prompts caused by user frustration. Coding agents can suggest changes, but humans decide which iteration is suitable for production.
