# Every Correction You Give Your Agent Is Data

Brian Douglas, Paper Compute | AI Engineer CODE 2026 | 12:48

Source: https://www.youtube.com/watch?v=vg3JdjGuJwM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/every-correction-you-give-your-agent-is-data
Published: 2026-11-10
Tags: agent-skills, agents, coding-agents, evals, observability, tracing

## TL;DR
- Agent sessions can preserve the moments when experienced engineers reject plausible code choices.
- Labels such as pushback, golden sessions, and regressions can turn session traces into review examples.
- Brian recommends creating a skill and evaluation from captured traces before attempting model adaptation.

## Summary
Brian Douglas argues that corrections from human engineers contain knowledge that coding agents lack. His example is Rust code that compiles but violates decisions about API structure, service duplication, or existing libraries. Because he does not have enough Rust experience to challenge the agent, the objections arrive later during pull request review. At Paper Compute, he captures agent sessions, stores them centrally, and labels moments such as expert pushback, successful sessions, and regressions. He then uses those traces in a Twin Review prototype that comments on a pull request in the style of a teammate. The prototype is private because its data is sensitive. Douglas also describes turning traces into skills and evaluations, then experimenting with specialized fine-tuning and direct preference optimization. His practical advice is to start with the traces and tools already available, create a skill first, and use evaluations to test whether it reflects the team's coding conventions.

## Key ideas
### Agent-generated code can miss decisions that compilation cannot detect
[00:01](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=1s)
Douglas describes writing Rust through agents even though he has not professionally written Rust. The generated code can compile and still violate decisions made by his team about API structure, interactions between parts of the codebase, and whether a service should be duplicated. Since he lacks enough Rust knowledge to challenge those choices during the session, he often accepts the output. The disagreement appears later when a human teammate reviews the pull request. The gap is about the team's conventions and design knowledge, not whether the compiler accepts the code.

### Session capture preserves the context around agent work
[01:47](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=107s)
Douglas introduces Tapes, an open-source Apache 2 project he uses to collect agent session data, while stressing that the same idea can be implemented with other telemetry collectors, session capture services, or agent harnesses. He also describes Stereos as a runtime for agents and reproducible environments. His setup records sessions and the signals from those environments. He has used parallel agents for defined work such as documentation, test writing, and code cleanup. The point is to retain what happened during agent use instead of leaving the reasoning inside a local harness.

### Expert pushback is a label for knowledge the agent does not have
[03:43](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=223s)
In Paper Console, Douglas can centralize sessions from himself and teammates and label individual sessions. He focuses on sessions where Matt pushes back on Rust or Go code. Those objections often encode decisions the agent could not infer, such as the team's API organization or existing services and libraries. Douglas calls this a useful signal because the person objecting has context that the agent lacks. He also notes that his own reviews may miss these problems, since he usually accepts the generated Rust code and does not know enough to challenge it before a pull request reaches human review.

### A labeled trace collection can support a simulated teammate review
[05:59](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=359s)
Douglas collects labeled sessions and runs a command-line tool called Twin Review against a pull request. The prototype uses traces where Matt pushed back on Go or Rust code and asks for comments as if Matt were reviewing the pull request. Douglas says this data comes from agent traces rather than GitHub review data. He connects the idea to a change in where review happens: teams increasingly ask agents to review code inside their harnesses, away from the shared codebase and without much written commentary. He later corrects himself and says Twin Review is not open source because the data is sensitive.

### Labels should include failures as well as successful examples
[08:06](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=486s)
Douglas says he looks for pushbacks, apologies, golden sessions, and regressions. Golden sessions are marked as sessions that deserve attention, while regressions capture cases where agents repeatedly fall over. One recurring problem is that an agent reinvents libraries that already exist, including mock libraries for testing. An engineer familiar with the codebase may reject that choice immediately, while someone without the same context may accept it. These labels give the team examples to use in a local review harness, a skill, or later model work.

### Skills and evaluations are a more practical starting point than training
[09:32](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=572s)
Douglas recommends starting by turning captured traces into skills. His local Tapes setup uses a Postgres database, so a team does not need Paper Console, which he says is waitlisted. After creating skills, the team can create evaluations that check whether the skill matches its coding practices. He says this path can identify whether Rust code follows the organization's conventions. Although he wrote about training a model instead of writing a skill, his recommendation is to write the skill first because it is likely to provide more value today.

### Model adaptation adds compute and hardware constraints
[10:48](https://www.youtube.com/watch?v=vg3JdjGuJwM&t=648s)
Douglas describes specialized fine-tuning as adding notes in the margin of an existing model, where the notes are weights or adapters. He says he used a 4B model and quantized it for smaller GPUs. He also tried direct preference optimization, which required more compute and did not work for him with a small 4B model. In his explanation, DPO needs a much larger model and more hardware. He used services including Fireworks and Brev during these experiments, but his closing advice remains to collect traces and create skills before taking on model adaptation.

## Notable quotes
- "The one thing I've been doing with this little setup of tapes and stereos is I've been building self-healing infrastructure." (02:34)
- "Whenever he pushes back, it's a good signal that he has knowledge that the agent does not." (05:01)
- "My recommendation is probably write a skill first. You can probably get most mileage out of a skill today." (10:16)
- "If you don't know about specialized fine tuning, this is like writing notes in the margin of a book." (10:48)

## Tools & references mentioned
- Paper Compute
- Tapes
- Stereos
- Sweeper Agent
- Daytona
- Anthropic
- Addy Osmani
- Rick Rubin
- The Creative Act: A Way of Being
- Beastie Boys
- Codex
- ChatGPT
- Claude
- Paper Console
- Twin Review
- Fireworks
- Braintrust
- Greptile
- Postgres
- Kubernetes
- Direct Preference Optimization
- Brev
- NVIDIA

## Who should watch
- You are building coding-agent workflows and need to retain the reasoning behind human corrections.
- Your team reviews agent-generated code after the session, but those reviews are not being collected as reusable examples.
- You are deciding whether to build a skill, an evaluation, or a fine-tuned model from your existing agent traces.

## Editor's note

Brian Douglas says Rust code can compile while violating a team's decisions about API structure and existing libraries, with the disagreement appearing during pull request review. Kitaru lets a person read recorded sessions and write notes about those moments; the notes become cohorts of runs that can capture the team's actual review judgment.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Let Your Agent Cook: Using Skills to Evaluate and Improve Your App](https://aietalks.com/talks/let-your-agent-cook-using-skills-to-evaluate-and-improve-your-app) (Ankur Duggal, Arize AI, 36:04)
- [Your Agent Fix Just Broke Something Else](https://aietalks.com/talks/your-agent-fix-just-broke-something-else) (Sofia Jakovcevic, Arize AI, 19:25)
- [Your AI Agent Is Hiding Its Mistakes From You](https://aietalks.com/talks/your-ai-agent-is-hiding-its-mistakes-from-you) (Ben Holmes, Warp, 19:20)
- [Respect The Process](https://aietalks.com/talks/respect-the-process) (Andrew Dumit, Watershed Technology Inc., 16:43)
- [An Agent That Fixes Its Own Mistakes](https://aietalks.com/talks/an-agent-that-fixes-its-own-mistakes) (Saloni Gandhi & Ziwei Guo, LinkedIn, 38:50)
