Braintrust users run evals frequently, but most evaluation work still ends with a person looking at a dashboard and deciding what to change.
2
Loop is an agent inside Braintrust that can optimize prompts, datasets, and scorers, including complex agents.
3
Ankur Goyal says recent frontier models, especially Claude 4, have made automated eval improvement much more practical.
Summary
Ankur Goyal argues that evals have become common in AI development, while the work around them remains manual. Braintrust users run anything from nearly 13 evals a day on average to more than 3,000 a day at some companies, yet the usual workflow still asks people to inspect a dashboard and decide how to change prompts or code. He introduces Loop, an agent built into Braintrust that can improve prompts, datasets, and scorers. Loop shows each proposed change beside the original data or prompt, or it can run in an automatic mode. Goyal says Braintrust has tested frontier models for eval improvement every quarter for two years, and that Claude 4 was a major improvement over the previous leading model. He expects eval work to change substantially over the next year and asks users to try Loop and provide feedback.
Braintrust customers already run evals at a high frequency
Goyal says Braintrust has spent almost two years working with companies that build AI products. The average organization that signs up runs almost 13 evals a day. Some customers run more than 3,000 evals daily, and some advanced companies spend more than two hours in Braintrust each day working through them. These examples show that evals are already part of regular development for some teams. Goyal presents this adoption as a contrast with the limits of the current workflow, since frequent eval runs still leave people doing much of the interpretation and follow-up work by hand.
The standard eval workflow still depends on a dashboard and manual decisions
Even companies building highly automated AI products and agents usually finish an eval by looking at a dashboard. The person running the eval then has to decide what to change in the code or prompts so the result improves. Goyal says Braintrust has a useful dashboard, but the basic interaction remains limited: inspect the result, leave the product, and work out the next edit. His criticism is about the amount of manual reasoning that follows an eval run. The eval can tell a team that something changed or performed poorly, but it does not usually carry the work through to a better prompt, dataset, or scorer.
Loop uses an agent to improve prompts, datasets, and scorers
Goyal introduces Loop, an agent built into Braintrust. It can automatically optimize prompts, including prompts for complex agents. It also helps create better datasets and scorers. Goyal treats these as connected parts of eval work, since strong evaluation depends on the prompt being tested, the examples used to test it, and the method used to score its output. Loop is presented as a way to move from simply inspecting eval results to making proposed changes inside the same product. The feature was available to Braintrust users through a feature flag at the time of the talk.
Recent frontier models made automated eval improvement more capable
For two years, Braintrust has run quarterly tests on frontier models to see how well they can improve prompts, datasets, and scorers. Goyal says the models were not very good at these tasks until very recently. He describes Claude 4 as a breakthrough and says it performs almost six times better than the previous leading model in their testing. This improvement is the reason Loop became practical enough to ship. The product can use Claude 4 by default, while also allowing users to select another model they can access, including OpenAI models, Gemini models, or a model they built themselves.
Loop keeps proposed edits visible inside the eval workflow
Goyal says Braintrust learned from users that it matters to inspect data and prompts while working on them. Loop therefore does not hide its changes behind an automated process. When it suggests an edit to data, a new scoring idea, or a prompt change, the product shows the suggestion side by side in the interface. Users can review what changed before applying it. There is also an option for people who want less supervision: a toggle labeled "just go for it" lets Loop optimize without requiring the same level of review. Goyal says this automatic mode works well.
Goyal expects eval work to change substantially over the next year
Goyal recaps that evals have been an important part of building strong AI products, but evaluation itself has been extremely manual. He expects the newest frontier models to change that over the following year. Braintrust plans to incorporate these model capabilities into its product, with Loop as an early example. He asks current and new users to try the product, test Loop, and provide feedback, while acknowledging that the team still has a lot of work to do. He also says Braintrust is hiring for interface, AI, and infrastructure work related to this problem.
"Every time it suggests an edit to your data or it suggests a new idea for scoring or it suggests an edit to one of your prompts, you can actually see that side by side directly in the UI."03:38
Who should watch
You run frequent evals but still spend much of your time deciding what to change after reviewing the results.
Your team is building an AI product and wants one workflow for improving prompts, datasets, and scorers.
You are evaluating whether model-assisted optimization is ready for your development process and want to inspect proposed changes before applying them.
