# Fighting AI with AI

Lawrence Jones, Incident.io | AI Engineer Europe 2026 | 17:29

Source: https://www.youtube.com/watch?v=L2r6vLlLgs8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/fighting-ai-with-ai
Published: 2026-05-17
Tags: agents, coding-agents, debugging, evals, multi-agent

## TL;DR
- Incident.io uses AI SRE investigations that run hundreds of telemetry queries and prompts, which makes human debugging too slow.
- A small eval CLI lets coding agents edit large YAML test suites and follow a red-green cycle that checks for regressions.
- Converting debugging UIs into downloadable file systems lets coding agents trace failures through prompt hierarchies, while parallel agents and clustering reveal fleet-wide failure patterns.

## Summary
Lawrence Jones describes how Incident.io uses AI to debug and improve its AI SRE product. The product investigates incidents by querying logs, metrics, traces, historical incidents, and code, but its hundreds or thousands of prompts make it difficult for a human to find the source of a wrong root cause analysis. The team built three internal patterns. An eval CLI gives coding agents focused access to large YAML test suites, allowing them to add a failing case, change a prompt, and check for regressions. They also serialize investigation and chatbot debugging views into file systems that agents can inspect alongside the code base. Finally, they run agents in parallel across batches of investigations, then cluster the analyses to find repeated failure modes across customer accounts. Jones argues that internal debugging tools should be designed around the way coding agents work, especially their ability to search files and follow written runbooks.

## Key ideas
### AI investigations have become too complex for human-only debugging
[00:38](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=38s)
Incident.io's AI SRE runs an investigation at the start of an incident, querying logs, metrics, traces, historical incident data, and the code base. It may run hundreds of telemetry queries and sit behind hundreds or thousands of prompts. A human may need about an hour to understand the incident well enough to judge whether the report is correct. That makes it hard to evaluate performance across customer accounts, especially because each account has different systems and failure patterns. Jones's premise is that the internal tools used to debug these systems also need AI assistance.

### Evals work like unit tests, but production cases quickly become unwieldy
[03:59](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=239s)
Jones describes an eval as an AI unit test. It takes input data, runs a prompt, and applies grading criteria to decide whether the result passes. Incident.io stores its evals in YAML files next to its Go prompts. Production examples can be valuable because a complete incident may be needed to trigger a failure, but those examples can grow to around 2 MB of YAML and become difficult to understand or maintain. Coding agents also hit their context limits when asked to read and modify the full files directly.

### A small CLI gives coding agents focused access to large eval suites
[07:00](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=420s)
Incident.io built an eval tool CLI with operations such as listing test cases, editing one, replacing one, and adding one. This lets an agent work with individual parts of an eval suite instead of loading the whole YAML file into context. The team uses it in a runbook for coding agents. Given a prompt problem, the agent creates an eval that demonstrates the failure, modifies the prompt until the eval passes, checks that other evals still pass, and tries to consolidate the prompt so repeated fixes do not make it unnecessarily large.

### The hardest debugging problem is locating the right prompt in a large hierarchy
[08:51](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=531s)
The eval workflow is reliable when the engineer already knows which prompt needs to change. Production systems, however, contain many prompts, tools, and agents. Jones shows a chatbot hierarchy with roughly 10 agents and about 50 other elements, too large to fit clearly on one screen. An investigation is even more complicated: each visible step can expand into hundreds of prompts and tool calls. A small error deep in that structure can produce a completely wrong root cause analysis, while leaving the engineer unsure where the failure began.

### File systems make complex debugging data usable by coding agents
[10:47](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=647s)
Incident.io converts its debugging UIs into downloadable file systems. The team drops an interaction into a Claude Code sandbox and describes the bad behavior. The agent can inspect the prompt hierarchy, trace inputs and outputs, read text versions of traces, and use the application code to identify where a change should be made. The engineer can then ask it to make the modification and use the eval runbook to test the result. Jones says this bulk file-based context worked better for them than putting an MCP layer on top of the same information.

### Fleet-scale analysis needs parallel investigation and later clustering
[12:50](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=770s)
Incident.io runs thousands of investigations across hundreds of customer accounts each day in backtests. A rolled-up result such as an 86% accurate root cause analysis does not explain why performance changed or how to improve it. Their analysis pipeline downloads investigations into a file system and uses markdown playbooks to guide a coding agent. The pipeline first starts about 25 agents in parallel, with each agent analyzing one investigation. A later stage clusters those results by recurring failure type and produces a report about why the system performs well or badly for an account and what the team should change.

### Analysis files and code access make the workflow resumable and actionable
[14:34](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=874s)
Jones recommends storing each agent's analysis in files inside the downloaded data. This creates incremental progress, so the pipeline can resume instead of starting over. The analysis should also include the code that powers the AI system. When an agent finds a repeated problem, it can connect the behavior to a likely code or prompt change. Because the work remains in the coding session, the agent can make a pull request, and the team can use the eval red-green cycle to check whether the fix works.

### Internal AI tools should match coding agents' strongest interfaces
[15:51](https://www.youtube.com/watch?v=L2r6vLlLgs8&t=951s)
Jones's general advice is to prioritize debugging tools that coding agents can use directly. In his experience, file systems are especially effective agent context because an agent can grep through a bulk download and find the relevant details. He recommends writing AI runbooks for complex, repeatable analysis, then using subagents for parallel per-entity work. The same session can inspect the evidence, connect it to the code base, propose a change, implement it, and validate it through evals.

## Notable quotes
- "You need assistance to help you." (01:22)
- "So, what we ended up doing was we ended up creating a small CLI tool that we call eval tool." (07:00)
- "Can we just download all of the UI that we have as a file system?" (10:47)
- "File systems are exceptionally good agent context." (16:23)
- "Anytime you are performing complex analysis, look at creating an AI run book for it instead." (16:34)

## Tools & references mentioned
- Incident.io
- Netflix
- Etsy
- Skyscanner
- Go
- Claude Code
- Codex
- Anthropic
- MCP

## Who should watch
- You are building an AI product with many prompts, tools, or agents and cannot reliably find which component caused a bad result.
- Your eval data lives in large files that coding agents cannot read or edit effectively.
- You run backtests across many accounts and need failure patterns that a single aggregate score does not reveal.

## Related talks

- [Taming Rogue AI Agents with Observability-Driven Evaluation](https://aietalks.com/talks/taming-rogue-ai-agents-with-observability-driven-evaluation) (Jim Bennett, Galileo, 16:15)
- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)
- [Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents](https://aietalks.com/talks/break-it-til-you-make-it-building-the-self-improving-stack-for-ai-agents) (Aparna Dhinakaran, Arize, 14:25)
- [Designing Agents (The Floor Is the Frontier)](https://aietalks.com/talks/designing-agents-the-floor-is-the-frontier) (Ben Hylak, Raindrop, 19:46)
- [The Agentic AI Engineer](https://aietalks.com/talks/the-agentic-ai-engineer) (Benedikt Sanftl & Burak, Mutagent, 34:50)