A reliable AI agent needs an eval suite, useful tools, suitable context, and a repeatable way to inspect regressions.
2
A coding agent can improve a product agent by forming one hypothesis at a time, testing it, keeping improvements, and rolling back regressions.
3
Feedback from real users can be clustered into failure modes, checked by subject matter experts, and turned into fixes and new regression cases.
Summary
Alfonso Graziano describes building production AI agents with a second AI system that writes and changes their code. The coding agent improves a product agent by running a golden-data eval suite, reading reports and traces, proposing one hypothesis at a time, and keeping or rolling back each change. In one simple example, the agent's pass rate rose from 18% to 83% in about 10 iterations. A production agent improved from 67% to 86% without changing the evals to make the score look better. Graziano also shows how live user feedback can reveal failures that the eval suite missed. Traces, comments, and expert annotations are clustered, reviewed, and converted into fixes or new eval cases. Human judgment remains part of triage and sets boundaries for what the coding agent may change. The whole workflow depends on a harness with specs, tests, evals, context, observability, and governance.
An agent is an LLM connected to tools and context inside a loop
Graziano starts with a simple model of an agent: an LLM is the brain, tools provide capabilities, context supplies information, and an agentic loop coordinates the work. Production systems also need observability and evals because the basic loop does not show whether the agent is behaving as intended. The problems include non-determinism, latency, cost, and hallucinations. He separates failures on a controlled eval suite from failures on live data, where real users bring wider and messier expectations.
A golden dataset turns uncertain agent behavior into something teams can test
A golden dataset is built with subject matter experts and records the input an agent should receive alongside the expected output. That output can be a value such as text or a number, but it can also describe behavior, such as calling a particular tool with a parameter or following a retrieval-then-update sequence. Scorers examine the dataset with an LLM and produce an accuracy measure. This gives the team a baseline, a way to spot regressions, and a record of what to improve.
Most early eval failures point to missing tools, instructions, or retrieval paths
Graziano's simple agent passes only 18% of its evals because basic questions can be answered from the model's training data while the rest require capabilities it does not have. An agent that must browse or fetch pages will fail if those tools are absent. Other failures come from a system prompt that lacks domain instructions or from context retrieval that cannot find the needed information. Improving the agent can therefore involve adding secure retrieval tools, changing the prompt, or correcting the tool contract.
A coding agent can improve a product agent through repeated experiments
Graziano applies the idea behind Andrej Karpathy's AutoResearch to agents. His AutoAgent loop runs evals, changes code, tries new system prompts, creates tools when needed, and checks the result again. A simple agent improved from a baseline of 18% to 83% in about 10 iterations. The approach also improved some evals on an already human-optimized production agent by 10%. The coding agent can find changes that the human team did not discover.
Branches, reports, and rollback keep optimization changes inspectable
Each optimization iteration starts on a new branch. The coding agent reads a baseline report, creates a hypothesis, changes the product agent, runs the eval suite, and writes a reports.md file. It also updates a shared memory file so later iterations can use earlier findings. An improvement becomes the starting point for the next branch. A regression or failed experiment is rolled back. The resulting change log lets humans inspect promising hypotheses even when the implementation did not work.
Live feedback reveals failure clusters that should become regression tests
For live data, the team collects traces from users, including tool calls, response time, token use, inputs, and outputs. Users can give thumbs-up or thumbs-down feedback with comments, while subject matter experts can annotate traces directly. The coding agent clusters failures, performs root-cause analysis against the trace and code, and proposes fixes. Experts then validate and triage the clusters as fixes to do now, fixes to postpone, or issues to discard. Confirmed failure modes are added to the golden dataset and eval suite.
A harness gives the coding agent enough structure to change code safely
Graziano calls the surrounding environment Harness Engineering. The harness gives the coding agent constraints, tasks, feedback loops, and governance. Every failure mode can become a specification for expected behavior. Quality gates include linting, unit tests, evals, and LLM code review. Context engineering gives the agent the files and information it needs. Observability shows what happened in production. Without those pieces, the coding agent cannot reliably judge or correct its own changes.
"Once we give enough context and once we give the coding agent the ability to test its changes against some regression tests, then everything becomes way, way easier for the coding agent."27:18
Who should watch
You are building an agent whose behavior changes across prompts, tools, retrieval, and code, and need a process for improving it without guessing.
Your eval suite gives you scores, but production users still report failures that the controlled tests do not cover.
You want a coding agent to make changes while keeping humans in control of specifications, permissions, triage, and rollback.
