Evals replace intuition-based shipping decisions with measured comparisons across datasets, tasks, scoring systems, and experiments.
2
In a coding-agent test on Microsoft's TypeScript Go repository, agentic search and vector search reached the same accuracy.
3
Vector search cost about four times as much because incomplete code chunks caused the agent to keep searching for context.
Summary
Jess Wang explains how evals help teams make decisions about AI systems with evidence instead of impressions. She breaks an eval into a dataset, a task, a scoring system, and experiments, then describes how production traces can feed new datasets and improvements. Her example compares agentic search with vector search for a coding agent. The dataset came from merged fix PRs in Microsoft's TypeScript Go repository. Claude generated bug-fixing tasks from the diffs, and Claude Code had to locate the buggy logic using either repository tools or vector retrieval. Both approaches reached the same accuracy when scored against the repository's test suite. Vector search cost about four times more because retrieved chunks often lacked imports, callers, tool calls, and other surrounding context. The agent repeatedly searched for the missing connections. Wang is open about the limits of the experiment, including its small dataset, weak vector implementation, and lack of repeated trials.
Evals turn shipping decisions into measurable questions
Wang contrasts shipping based on a few prompts or an engineer's opinion with reporting test results across many cases. An eval can answer which model fits a use case, how performance changes across languages or programming languages, whether costs are acceptable, and whether a system has regressed. She gives the example of an OpenAI model update in April 2025 that was intended to be more helpful but became too agreeable and less truthful. An eval could have exposed that change.
An eval has four parts that produce comparable experiments
The four parts are a dataset, a task, a scoring system, and experiments. The dataset can contain standard cases, edge cases, and known failure modes. The task defines how the system should respond, including the system prompt and selected model. The scoring system defines what counts as good or bad, using methods such as deterministic checks, an LLM judge, or human review. Each configuration produces an experiment that can be compared with others, including changes in individual test rows.
Production observability feeds an eval improvement loop
Wang describes a loop that starts with an AI application and its production logs. A team can sample real interactions into a dataset, define a task and scoring system, run several evals, and use the results to change prompts or code. The updated system goes back into production, where its new logs provide more material for the next cycle. Braintrust also supports querying traces and experiments in natural language, alongside its SDK, CLI, and MCP server.
Wang calls evals a team sport because the work includes collecting real-world data, labeling it, forming hypotheses, and analyzing results. AI engineers make code changes and bring data into the platform. Product managers help define what success means and adjust prompts. Subject matter experts can provide labels or ground truth in areas such as insurance, medicine, and law. Data analysts study the resulting measurements.
Vector search retrieves semantically similar code chunks
Vector search converts code or text into embeddings that capture semantic meaning, then stores them in a vector database such as Qdrant or Pinecone. A query like 'where does the database logic in my code base live?' returns chunks that are semantically close to terms such as database and logic. Wang illustrates the idea with animal embeddings, where wolf, dog, and cat are close together, while fruit embeddings such as banana and apple form a separate group.
Agentic search follows code relationships with repository tools
Agentic search gives the language model tools such as grep, find, ls, cat, and other bash commands. The model can search for a function, open its file, read the surrounding code, follow a function call into another file, and continue until it has enough information. Wang describes this as closer to how a person investigates a codebase because the search can follow relationships between definitions and callers rather than relying on one retrieved chunk.
The coding-agent eval used real fix PRs and binary test scoring
Wang built the dataset from merged pull requests with 'fix' in their titles in Microsoft's TypeScript Go repository. She checked out the parent commit, where the bug still existed, then used Claude to generate a bug task from the difference between the buggy and fixed code. The agent had to locate the buggy logic. The experiment used about 20 rows. A task scored 100% when the repository test suite passed and 0% when it failed.
Agentic search and vector search reached the same accuracy, but vector search cost about four times more. Retrieved chunks often omitted imports, tool calls, or calling code, so the model had to search repeatedly for the missing context. In one row, the vector agent made 26 searches without piecing together how three functions interacted. Wang describes vector search as providing proximity to the right code, while agentic search supplied more of the connective tissue needed to understand the logic.
The result needs more trials and a stronger vector baseline
Wang lists several limits to the experiment. The vector search implementation was relatively weak, the dataset contained only about 20 rows, and the study did not run multiple trials per task. Repeated trials would make the score more trustworthy because language-model behavior is nondeterministic. She also suggests expanding the dataset to other repositories. The purpose was to demonstrate an end-to-end eval rather than establish a final comparison.
"Vector search gave a lot of proximity to the correct code, but agentic search had more of the connective tissue to connect the actual logic of those chunks of code."15:56
Who should watch
You are deciding between retrieval strategies for a coding agent and need to compare accuracy with runtime cost.
Your team ships AI features from prompt checks or manual testing and needs a concrete eval structure.
You are instrumenting agent subprocesses and want to understand why parent and child traces can become disconnected.