Agent quality depends on the signals that guide each loop and the verifiers that judge its results.
2
Teams should capture evals, judges, skills, prompts, tools, and human judgment in portable, versioned agent recipes.
3
Progress should be measured by the value produced for the compute used, then validated with offline evals and real users.
Summary
Roland Gavrilescu presents a blueprint for autoresearch built around three ideas. The loop is the product: an agent acts, observes signals, produces work, and gets judged by verifiers. The output then feeds another loop that can improve the system. System distillation is the moat: teams should turn lessons from each run into portable, versioned recipes containing evals, judges, skills, prompts, tools, model profiles, and human judgment. Introspection's early Pi recipes release applies this approach to the Pi harness and Harbor evals. The final measure is valued work per watt. Gavrilescu explains how to turn a maker's taste into evals with a talent-sourcing example. Trace patterns reveal unwanted behavior, humans calibrate the resulting judges, and production A/B tests show whether users value the change. The talk argues that agent products improve through repeated cycles of observation, judgment, codification, and validation.
Gavrilescu says agent development has moved from training models to building harnesses and then to building loops. He uses an OpenClaw car-buying example: the agent searched Reddit for prices and inventory, contacted dealers, made them compete, checked whether the price was right, and locked in the purchase. The loop worked because each step produced information for the next one. He connects this to OODA loops, where an agent acts, observes, and reacts in a fast-moving environment. Strong signals guide the worker, while a verifier checks whether the result is actually correct.
A second loop turns results into continuous improvement
The first loop produces artifacts that can become inputs to a second loop. Gavrilescu says the quality of the signal affects the loop's success rate, while the verifier calibrates whether that success is real. Running another loop over the first loop's output allows the system to improve continuously. The useful output can include failure patterns, repeated behavior, user frustration, and changes that worked. This gives the system a way to learn what should be changed instead of repeatedly starting from the same base harness.
Agent recipes preserve the system outside any provider
Each loop generates information about harnesses, profiles, evals, models, resources, tools, and the environment. Gavrilescu proposes storing this information in portable, versioned agent recipes. A recipe can contain evals, tweaks, human judgment, skills, prompts, and other decisions that emerge while the agent operates. He compares this with data recipes in reinforcement learning, where teams update the recipe to address hallucinations or reward hacking. The recipe belongs to the company, can live in a Git repository, and remains usable across models and providers.
Introspection's early release, Pi recipes, is built around the Pi harness and Harbor for evals. Gavrilescu describes a recipe as more than a harness or model choice. It records how the system was developed and why particular signals, tools, profiles, and evals were selected. Agents manage changes in versioned repositories, while the owner supplies the higher-level taste that the agents must learn. Failure patterns can become judges and evals. Repeated behavior can become skills and prompts. The aim is a reproducible agent system that another person can use with the same underlying judgment.
Gavrilescu says teams should optimize for the value produced per watt of compute. He describes a progression from building a product, to building evals for that product, to building models based on those artifacts. A team only learns where the frontier is by running systems in production. After reaching that frontier, it must reduce the cost of producing the same value. This applies beyond coding, including customer support and legal research. The work therefore needs two measurements: whether the output is valuable and whether the economics make sense.
Taste becomes an eval through traces and human calibration
The talk's talent-sourcing example starts with web search, LinkedIn, subagents, and a recruiter system instruction. The team examines traces to find repeated behaviors and user frustrations. For example, an agent may contact many big-tech employees when the recruiter wants hidden candidates from GitHub. That pattern becomes a judge or eval that checks future trajectories. The agent can generate the eval, but a human must confirm the judgment. The human does not need to write the eval. Their role is to decide whether the encoded preference matches the maker's taste.
Production tests decide whether a recipe change is worth promoting
Offline evals can produce a candidate recipe, but the final test happens with users. Gavrilescu proposes A/B testing the revised behavior, including a multi-armed bandit setup, to see whether users value the maker's preference. In the sourcing example, the question is whether users prefer avoiding obvious big-tech targets in favor of hidden gems. A change is promoted only after the production result supports the taste encoded in the eval. Repeating this process turns a person's judgment into an agent that can reproduce the same service for other people.