# Your AI Agent Is Hiding Its Mistakes From You

Ben Holmes, Warp | AI Engineer NYC 2026 | 19:20

Source: https://www.youtube.com/watch?v=pz74r-4dGfQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-ai-agent-is-hiding-its-mistakes-from-you
Published: 2026-11-06
Tags: agents, coding-agents, continual-learning, evals, observability

## TL;DR
- Agents can review their past conversations, score them against rubrics, and use the failures to suggest changes to skills and configuration.
- A useful agent trace includes the conversation, tool calls, external connections, and resulting code or pull request, because the agent can hide the workarounds it used.
- Warp runs a daily self-improvement agent that groups failed conversations and proposes pull requests, which engineers can review before merging.

## Summary
Ben Holmes describes a feedback loop for improving coding agents. The system collects agent conversations, tool calls, connected services, and outputs in one place. Scoring agents then review each conversation against rubrics such as redundant tests, verbosity, code quality, and efficiency. This matters because agents can recover from broken instructions without reporting the problem. Holmes gives an example where outdated testing instructions made an agent probe port numbers repeatedly, adding time without making the failure visible. Once failures are scored, a scheduled agent groups similar problems and suggests changes to skills or configuration. Holmes demonstrates Skill Doctor, which grades local Claude Code and Codex history, and shows Warp Factories proposing pull requests from its ongoing evaluations. He also argues for tracking scores over time, so teams can connect a merged skill change with a later improvement in agent performance.

## Key ideas
### A software factory also needs engineering work on its own machinery
[01:00](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=60s)
Holmes defines a software factory as an automated software development lifecycle. Agents can research and triage tasks, help write specifications, implement changes, capture screenshots, prepare pull requests, and run adversarial code review. Around that workflow is a second layer of work, which Holmes calls factory engineering. It improves the system that drives the agents. The talk focuses on agent skills and configuration. The proposed loop has agents review their past conversations, scoring agents grade the work against defined rubrics, and another agent inspect the failures for changes that could improve future runs.

### A shared trace must contain what the agent did outside the conversation
[03:24](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=204s)
Teams can collect local agent history from Claude Code, Codex, or similar tools, then copy it into a shared S3 bucket. A factory with shared runners or isolated Docker sandboxes may centralize the traces automatically. Holmes says the stored record should include the transcript, tool calls, file system calls, MCP connections, and the resulting output. Outputs can be code diffs or the pull request and commit that contain the change. Without these parts, later grading cannot see how the agent reached its result or what it produced.

### Agents can hide broken instructions by working around them
[08:01](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=481s)
Holmes says agents can be persistent enough to work around roadblocks without telling the user. In Warp's test harness, skills contained outdated instructions. The agent discovered the environment by trying several port numbers, then continued with the coding task. The team mainly noticed that the run took about two minutes longer than expected. A trace that includes tool calls makes this visible. An efficiency scorer can then judge whether the agent moved toward the solution or spent time on repeated exploration and failed calls.

### Simple rubrics give scoring agents a concrete standard
[08:57](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=537s)
Holmes shows a markdown skill definition for an LLM-as-judge scorer. The rubric describes the criterion, then labels what passing and failing behavior look like. For efficiency, it asks whether the agent worked toward the result or wasted effort on redundant steps and unnecessary back-and-forth. A score below the chosen boundary fails, while a score above it passes. He compares this with a school essay rubric and says teams can start with a few bullet points, then refine the definition as they learn what they want to measure.

### Conversation grading can connect a specific failure to a skill change
[10:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=641s)
A grading run can inspect every conversation or use random sampling to reduce token use. In Holmes's example, an agent grades a full-stack change across test coverage, verbosity, and efficiency. It fails the test-coverage criterion because the agent wrote many unit tests when the team's guidance preferred end-to-end tests. It passes verbosity because the pull request was clear and contained few code comments. It fails efficiency because the agent explored the test environment instead of receiving better instructions up front. Each result also gets an explanation that a later improvement agent can use.

### A scheduled improvement agent looks for patterns across failures
[12:51](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=771s)
Holmes describes grading conversations after they happen, often when a pull request is merged. Every 24 hours, a scheduled agent samples recent conversations, filters for failures, and receives the failure reports and explanations. It also gets access to the skills repository and code base. The agent looks for common causes across the failed runs and proposes a change. One example adds a coverage pyramid after redundant tests appear across multiple pull requests. This turns individual grading results into a possible update to the system's shared instructions.

### Skill Doctor makes the review loop available on a local setup
[13:47](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=827s)
Skill Doctor is a Warp skill that crawls through local Claude Code, Codex, and other agent conversations for a one-time grading run. Holmes shows it processing sessions in batches and producing reports for metrics such as efficiency and code quality. A second agent reads those reports and suggests changes to the skills. In his example, it recommends telling the end-to-end testing skill how to find an available port before starting the test. It also rewrites a changelog skill with more prescriptive language, reducing repeated manual corrections when an agent opens a pull request.

### Score history lets teams check whether an improvement actually helped
[16:14](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=974s)
Holmes argues that self-improvement should remain visible through a dashboard rather than only through hidden agent memory. Teams can track scores such as efficiency over time and mark when a skill change was merged. If the score changes after that point, the team has evidence about the effect of the update. In Warp Factories, continuously scored conversations feed an inbox of proposed pull requests. One merged change fixed missing configuration for spawning sub-agents in a triage skill, preventing repeated tool calls while the agent tried to work out how to create them.

## Notable quotes
- "Agents that look through their past work and figure out how to improve their own setup." (00:12)
- "You got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output which is something that you may want to grade for quality purposes." (05:38)
- "Agents these days are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it." (08:18)
- "It's more of an art than a science." (10:20)
- "By having scores in the loop with self-improvement, you can get that sort of view into things." (17:49)

## Tools & references mentioned
- Warp
- Warp Factories
- Skill Doctor
- Claude Code
- Codex
- S3
- Open Claw
- Docker
- MCP
- SonarQube

## Who should watch
- You run coding agents in cloud sandboxes or other unattended environments and need to find failed tool calls that the agent does not report.
- Your team has recurring problems with test volume, verbose code changes, inefficient exploration, or unclear agent instructions.
- You want a practical pattern for grading agent history and turning repeated failures into reviewable skill or configuration changes.

## Editor's note

Ben Holmes says agents can work around broken instructions without telling the user, leaving teams to notice only that a run took longer. Kitaru records every model call and tool result in a session, so engineers can inspect the hidden path and replay the failed run against a changed prompt, model, or code without touching real systems.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Your agent is blindfolded](https://aietalks.com/talks/your-agent-is-blindfolded) (Johan Lajili, Poolside, 09:58)
- [Your Agent Fix Just Broke Something Else](https://aietalks.com/talks/your-agent-fix-just-broke-something-else) (Sofia Jakovcevic, Arize AI, 19:25)
- [Building Self-Learning Loops for Your Agent](https://aietalks.com/talks/building-self-learning-loops-for-your-agent) (Fuad Ali, Arize AI, 38:55)
- [An Agent That Fixes Its Own Mistakes](https://aietalks.com/talks/an-agent-that-fixes-its-own-mistakes) (Saloni Gandhi & Ziwei Guo, LinkedIn, 38:50)
- [Agents That Self Improve While You Sleep](https://aietalks.com/talks/agents-that-self-improve-while-you-sleep) (Fuad Ali, Arize AI, 53:05)
