Notion spends about 10% of its time prompting and 90% examining evals, usage, and iteration results.
2
A useful eval combines a task, a data set, and scoring functions, with deterministic checks and focused LLM judges used together.
3
Production logs, user feedback, and human review should feed new examples back into the data set and improve future evals.
Summary
Sarah Sachs explains how Notion evaluates AI features while supporting more than 100 million users, many of whom work in languages other than English. Notion builds targeted data sets from logs, internal use, and product-specific examples, then uses deterministic checks and focused LLM judges to test changes. The team separates retrieval evaluation from generation evaluation because indexes change and can make results difficult to interpret. Carlos Esteban and Doug follow with a Braintrust workshop covering offline evals, production logging, online scoring, human review, and remote evals. The practical message is that prompt writing is only a small part of AI product work. Teams need to inspect outputs, maintain useful test cases, compare models and prompts, and turn production feedback into new evaluation data. The talk also covers playground and SDK workflows, multi-turn interactions, chained agents, custom models, and running complex local evaluations from the platform.
Notion treats evaluation and observability as the main work of scaling AI products
Sarah Sachs says teams spend roughly 10% of their time prompting and 90% looking at evals, iterating on evals, and examining usage in Braintrust. This prevents a feature from being judged by a single successful demo. Notion needs that discipline because it supports more than 100 million users, offers free trials for almost all AI products, and aims to ship newly released models to production in less than a day. The company also cares about the polished customer experience associated with Notion's design-driven product. Evaluation lets the team move quickly while checking that a feature works consistently for the users it is meant to serve.
Notion developed features in step with model capabilities instead of starting with today's agent products
Notion AI began with the AI Writer, which generated text inline, then expanded into Autofill, where an AI agent operates across database properties. The team later built retrieval and Q&A for connected workspaces, including embeddings and multilingual support, before adding search across external apps and uploaded attachments. Its newer products include meeting notes, enterprise search, and deep research. Sarah describes deep research as a move from fixed workflows toward agents that can reason about tools and spend longer on a task. She says it would have been naive to begin with the newest products because earlier models could not support them.
Evaluation data should start small, be close to real usage, and be structured before it grows
Notion creates targeted data sets from logs, internal usage, and handcrafted examples. Sarah recommends starting with 10 examples, checking that they have the right structure, and only then expanding. Notion once created large amounts of dummy data that was badly structured, which made the data difficult to use. Data specialists at Notion combine product management, analysis, and annotation work. Smaller companies can assign this work to a PM or an engineer who is close to the data. Sarah also says quality matters more than quantity when human evaluators are extracting insights, and thumbs-down feedback from development usage is brought into Braintrust.
Focused judges and deterministic checks reveal more than one broad score
Sarah distinguishes between a single judge prompt applied to an entire data set and separate prompts written for individual examples. Her preferred approach describes the rules for each expected output, such as answering in Japanese, using a required bullet format, or citing a particular page. This works well for search because the expected rule can stay current even when the index changes. Notion also uses deterministic checks, such as verifying that a tool call queries Jira or that an output uses the intended language. The team runs multiple judges at different scopes and examines failures instead of treating one score as a complete description of quality.
Retrieval and generation need separate evaluation paths when the index changes
Sarah says a changing search index can make an evaluation confusing because a query's correct answer may depend on what documents were available at the time. Freezing the entire index is technically difficult, expensive, and connected to changing permissions. Notion instead evaluates retrieval on its own by checking whether the right items were returned. It then freezes those retrieved results and evaluates the rest of the response as generation. If the needed answer was not present in the retrieved set, the sample is removed from that evaluation. This separation helps the team identify whether a failure came from search or from the model's later work.
Carlos Esteban defines an eval as a structured test of how well an AI system performs across scenarios. The task can be a single prompt, a multi-turn exchange, an agentic workflow, or a chain of prompts. The data set contains real-world examples or test cases, with optional expected outputs and metadata. The score applies logic to the result and can use an LLM judge or a deterministic heuristic. Offline evals use predefined data while iterating on prompts and models. Online evals score live production traces, helping teams monitor performance, diagnose problems, and capture feedback for later improvements.
LLM judges themselves need testing against human judgments
Carlos recommends using a stronger model to judge smaller models, while keeping each judge focused on one criterion. Teams should evaluate the scoring prompt with human-annotated examples and check whether its decisions match the human reviewers. Sarah describes using a scale in practice, although the team groups results below a threshold together and still has humans inspect failures. Notion also passes failures to another LLM to summarize the largest themes across thousands of samples. Sarah is candid that the scale is not well calibrated in their process, and that further calibration research has not yet justified the investment for Notion.
Production tracing turns user feedback into data for the next evaluation cycle
The Braintrust workshop shows how wrapping an LLM client can capture tokens, latency, cost, and nested calls. Teams can add online scoring rules to live traces, choose which spans to score, and sample only part of production traffic while they gain confidence in the metrics. Views can filter logs, such as responses below a completeness threshold. Users can submit thumbs-up or thumbs-down feedback with comments and metadata. Human reviewers can apply configured scores to logs or data-set rows. Those reviewed spans can then be added to data sets, connecting pre-production evals, production behavior, and future prompt or model changes.
"We spend maybe 10% of our time prompting and 90% of our time looking at evals and iterating on our evals and looking at our usage."02:03
Who should watch
You are building an AI feature and need a way to compare prompts, models, retrieval changes, or agent workflows before release.
Your production logs contain user feedback, but that feedback is not yet becoming evaluation data or changes to the product.
You need product managers, designers, engineers, or subject-matter reviewers to work on AI quality without putting every task inside application code.
