An eval platform grows from a loop, spreadsheet, and examples into a system for experimentation, production tracing, analytics, and team workflows.
2
Production traces provide the failure modes and real examples needed to design useful scoring functions and improve agents through repeated offline evals.
3
The hardest part is the data layer because agent traces are large, fast-moving, semi-structured, text-heavy, and subject to different query patterns.
Summary
Phil Hetzel argues that an eval platform starts simply, with a way to run an agent, collect input examples, and record scores in a spreadsheet. That setup helps teams begin, but it becomes slow and difficult to compare as agents change. A more mature platform needs experimentation for technical and non-technical users, sandboxed agent configurations, scoring functions tied to failure modes, and a playground. It also needs to connect offline evals with production observability. Real traces reveal how users interact with an agent, and those examples can feed back into offline testing. Hetzel says the resulting system is difficult because traces are large, high-volume, semi-structured, and full of text. It must support fast trace viewing, aggregate analysis, and full-text search. Access control, data masking, automatic tracing, multimodal files, and agent-driven workflows add more requirements. His central point is that the data platform is harder to build than the eval interface.
Hetzel recommends beginning with a spreadsheet, a loop over input examples, and a way to execute the agent. This gives a team visibility into how outputs change after each agent modification, with almost no barrier to entry. The limits appear as the project grows. A spreadsheet mostly documents experiments rather than supporting them, makes direct comparisons difficult, offers little analytics, and depends heavily on human scoring. It also excludes non-technical contributors who have domain knowledge and proximity to users. Recreating or appending results becomes a slow process each time the team evaluates a new version.
A custom UI improves access without creating real iteration
The next stage is a bespoke interface backed by a database instead of Excel or Google Sheets. A nicer UI can bring more people into the process and give the team better persistence for eval results. Hetzel compares this stage to a reporting tool because it still does not encourage much experimentation. Teams may have a for loop, a dedicated interface, and a database, but they are mainly recording and displaying results. The platform has become easier for specific users to access, while the underlying workflow remains focused on documentation.
Playgrounds let more users compare agent configurations
A more useful platform gives technical and non-technical users access to an agent configuration and a sandbox. In Hetzel's example, a user changes an agent's system instructions in a UI, compares two prompt configurations, and runs evals across both agent runs. The platform then surfaces technical and functional scores for comparison. He says playground features should support both SDK-driven workflows and UI-based experimentation. This is where the platform begins to connect changes in an agent with measurable behavior instead of merely recording what happened.
Hetzel says the best way to evaluate an agent is to identify the failure modes it can fall into and build scoring functions around those cases. Production trace data helps teams find those failure modes because it shows the agent handling real users and real usage. This changes the purpose of evals. Rather than testing only examples someone prepared in advance, the team can use observed behavior to decide what should be scored. The platform must therefore make production data available to the people designing and running offline evaluations.
Offline evals and production observability form a continuous loop
Hetzel describes observability and evals as one systems problem. Production traces show how the agent actually behaves, while offline evals let the team safely rerun examples and compare changes. The team can pull real interactions into an offline environment, improve the agent against those examples, and then observe the changed agent in production. He compares mature evals to rerunning production in a safe environment. This loop should continue for the lifetime of the agent. At this stage, the platform also needs online scoring and alerting against observability traffic.
Once a platform handles production traffic, agent traces create difficult storage and query problems. Hetzel says they are often semi-structured or unstructured, very large, and high velocity. Some spans can reach 10 or 20 megabytes, compared with a few kilobytes for a traditional span. The system needs one low-latency path so users can see traces quickly and another persistence layer for aggregate analysis. Full-text search adds another requirement because LLM traces contain so much text. These needs make the backend a substantial systems problem, even when the interface looks simple.
Hetzel expects eval platforms to expose unknown patterns in agent use through techniques such as topic modeling, so engineers do not have to inspect every trace manually. He also says platforms should be built for agents as well as people. A coding agent could query eval data, bring aggregate results into context, and change the evaluated agent during a coding session. That workflow is difficult when the backend only supports basic SQL access. Mature systems also need role-based access control, data masking, and centrally enforced tracing through an AI proxy or gateway.
Building an eval platform creates an ongoing ownership burden
Hetzel is direct about the cost of building an internal platform. Once a team creates it, someone has to manage it and keep expanding it as the industry changes. The work does not end when a UI or database is running. The team must support new data sizes, query patterns, users, scoring methods, and production workflows. The platform can also need multimodal support. Braintrust stores audio or video in object storage, references those files from traces, and displays them for review without requiring users to leave the platform.
"This is where the rubber starts meeting the road because the best way to perform evals is to really think about the failure modes that your agent can fall into and build scoring functions around those failure modes."14:09
Who should watch
You are running evals from a spreadsheet and need to understand what breaks as more teams and agent versions join the process.
Your team is designing an internal eval or observability platform and needs to plan for trace storage, search, scoring, access control, and production feedback loops.
You want to connect real user traces to offline testing instead of relying only on hand-written examples.
