Building AI agents needs an offline loop for testing and improvement, followed by an online loop that monitors production traces and feeds failures back into development.
2
An agent specification defines responsibilities, context, integrations, constraints, and success criteria before a coding agent implements it on a chosen platform.
3
Automated evaluation and diagnosis reduce manual trace review by turning production failures into new test cases, failure categories, and proposed fixes.
Summary
Benedikt Sanftl and Burak describe an Agentic AI engineer as a multi-agent system that builds and improves AI agents through two connected loops. The offline loop covers specification, implementation, evaluation, and deployment. The online loop monitors production traces, diagnoses failures, adds new evaluation cases, and generates changes that can be tested before deployment. They argue that manual review becomes a bottleneck as teams operate more agents. Their approach uses evaluator and diagnostics agents, coordinated by an orchestrator, to inspect trajectories, group failures by root cause, and produce remedies for a coding agent. The talk also explains why specifications should stay separate from implementation frameworks, why evals need actionable criteria, and how representative trace sampling can reduce the cost of diagnosis. The final section demonstrates a diagnostics workflow that produces an HTML report and a Markdown task definition for applying selected fixes.
Agent development needs connected offline and online loops
Sanftl and Burak separate agent development into an offline loop and an online loop. During development, the team iterates on an agent, tests it, evaluates it, and improves it. After deployment, the online loop monitors traces, diagnoses issues, and feeds the findings into optimization. The older process handled these steps manually: someone changed the agent, generated samples, reviewed results and traces, shipped a version, and ran A/B testing. Human review and building time then limited how many cycles a team could complete. The proposed system automates parts of both loops so teams can run more cycles in the same time.
The lifecycle moves from specification through optimization
The lifecycle begins with a specification that defines an agent's responsibilities, functions, and decisions under different conditions. A build stage turns that specification into an agent in a chosen harness, framework, or coding-agent environment. Evaluation then checks whether the agent meets its criteria, similar to unit tests for software. After shipping, monitoring detects issues and can trigger diagnosis based on trace volume or scheduled jobs. Diagnosis groups failures and traces them to causes. Optimization creates targeted changes or mutations for those failure modes. The cycle returns to evaluation, and a passing version can be deployed again.
A stable specification lets teams change implementation platforms
The specification is kept separate from implementation details. A coding agent can use it to create an initial version on the target platform chosen by the team. This matters because agent frameworks and harnesses change quickly, and a team may hit a limitation in its current framework. Burak says that after building agents for three years, they have sometimes needed to wait for an underlying framework to remove a bottleneck. Keeping the requirements isolated allows a team to choose another harness when needed. The talk mentions Hermes and Deep Agents as examples of newer frameworks in this changing area.
Evaluation suites grow from discovery and production evidence
An evaluation suite contains both criteria or metrics and the data cases used to test an agent. Teams can begin by working with domain experts, historical data, or synthesized examples. Burak says they cannot predict the complete suite at the start. User feedback and production failures add representative edge cases and difficult cases over time. The resulting suite lets the team run the agent against known cases and see exactly where it fails. Automated evaluation matters because manually inspecting a dataset of 200 items through dashboards and logs makes each experiment slow. An evaluator agent can sift through this work in the background.
Agent evaluation must inspect the full trajectory, not only the final answer. The checks include whether the agent had complete context and whether each tool returned the right output, since one bad tool result can cause a wrong final response. The harness itself can also affect behavior. Burak says score-based LLM judges are less useful when their rubrics do not point to a fix. Binary criteria can give a clearer action when they fail. LLM judge systems also need calibration because the same judge may score the same problem differently across runs. Without controlling that variance, comparisons between agent versions are difficult.
Diagnosis turns repeated failures into learned indicators
Once an agent runs in production, diagnostics identify failure modes and group them by root cause. Causes may include a prompt section, missing tools, or malfunctioning tools. The findings can produce new evaluations that detect the problem and proposed remedies that address it. At first, diagnosis often requires reading traces closely. Over time, each failure mode can acquire code-checkable indicators, such as specific content or a known tool-call sequence. Those indicators allow the system to find problems without reading every trace. When an agent produces millions of traces, reviewing all of them can cost more than running the agent, so diagnostics should select representative samples through segmentation.
The product coordinates evaluator and diagnostics agents
Mutagent presents its system as a set of agents connected by an orchestrator that runs in the user's coding environment. Two agents are in research preview: an evaluator agent that helps build an evaluation set and a diagnostics agent that analyzes production traces. Connectors can bring in traces, incidents, ticket reports, or Slack reports. The system can target different frameworks and outputs, including a GitHub pull request, changes to agent Markdown files, or a managed service. In the demo, a diagnose command starts root-cause analysis for an agent or skill and retrieves traces from sources such as Langfuse, local cloud transcripts, or exported JSONL.
The diagnostics report leads to selected fixes for a coding agent
The diagnostics workflow produces an HTML artifact with details about the agent, its tools, and its harness or framework when code access is available. It filters trace samples in multiple tiers, first using a small portion to detect obvious problems, then focusing on particular signals or failure modes. Users can also guide the search with a reported issue. The report shows detected issues and their frequency, explains each failure mode, and gives a recursive why chain for its origin. It lists assumptions so users can correct conclusions made without full code access. Users select remedies, then receive a Markdown task definition that can be handed back to their coding agent.
"If you have millions of agent traces and to read all of these it actually costs more than the execution itself."21:00
Who should watch
You are building several AI agents and manual review of samples, traces, and production failures is slowing development.
Your agent framework may change, and you want requirements that can be implemented on different harnesses or platforms.
You need an evaluation process that turns production failures into test cases and concrete tasks for a coding agent.
