# Your PR Says the Tests Passed. Prove It.

Stephen Chin, Neo4j & Cassandra Chin, Tracetoon creator | AI Engineer CODE 2026 | 23:01

Source: https://www.youtube.com/watch?v=gbeuC0ud1p4
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-pr-says-the-tests-passed-prove-it
Published: 2026-11-10
Tags: cost, evals, observability, tracing

## TL;DR
- Tracetoon reconstructs agent activity across sessions so pull requests can explain what changed, why it changed, and what evidence supports the claims.
- A passing-test claim is unreliable when the recorded commands do not show that the submitted code was tested in the required sequence.
- Relate builds a local evidence graph before drafting, allowing a stronger writer to use a compact brief while cheaper reviewers check claims and reduce token use.

## Summary
Stephen Chin and Cassandra Chin examine why AI-generated pull requests can contain fluent but unsupported claims. Tracetoon gathers context from agent sessions, code changes, commands, and audit trails, then uses that material to write and review a pull request. Their example exposes a serious error: a PR claimed that tests passed even though the trace showed incompatible commands running at the same time. Repeated revisions improved the prose but consumed tokens and still allowed mistakes. The newer design uses Relate, an embedded Rust graph database, to connect a change with its rationale, command history, timing, and tested code. This produces a connected evidence brief before writing begins. A higher-quality model writes the PR, while less expensive reviewers check claims against the brief. Their PR Bench comparisons showed better judged descriptions and roughly half the token usage for this project. The speakers present this as an engineering pattern, not a guarantee for every codebase.

## Key ideas
### Tracetoon began as a way to recover the work hidden behind AI-generated code
[00:00](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=0s)
Cassandra Chin built Tracetoon after seeing pull requests rejected because their descriptions contained incorrect information or were hard to read. The submitted program shows the final artifact, while the prompting and agent activity that produced it usually remain outside the repository. Her earlier Show Tell work focused on tracing how students created programs. Tracetoon applies that same idea to pull requests by collecting agent context, code information, and session history. It can also add a comic that explains the change, although the engineering purpose is to improve the accuracy and readability of the PR.

### Reading one agent session is insufficient because context becomes compressed and inaccurate
[04:20](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=260s)
A coding agent often writes a pull request by reading its own session log. Stephen Chin says the agent's context window gets compressed as the session continues, so the resulting description can become less accurate. Tracetoon instead searches multiple sessions on the developer's computer to reconstruct what happened across the branch. That evidence can include the working directory before a command changes directories, the reason for a command, and information about possible test-data leakage between repositories. This gives the PR more than a line-by-line explanation of the diff.

### Repeated research and revision can improve a PR while still leaving factual mistakes
[07:24](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=444s)
The original Tracetoon flow finds traces for a branch, researches them, reviews the draft, adds context, and repeats the process until it has enough evidence. Cassandra says it produced better descriptions and fewer inaccuracies than working without Tracetoon, both subjectively and through PR Bench. It still sometimes made mistakes during later revisions, when the system tried to rewrite with more context. One example claimed that no leader passing run was recorded, even though the trace showed bun test running at the same time as bun x tsc. Those commands had to run sequentially for the test to pass, so the claim did not match the actual execution.

### Using cheaper writers reduced cost but also reduced the quality of the pull request
[10:23](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=623s)
The speakers tried replacing expensive writing models with cheaper ones to address token spending. The result was worse quality rather than a useful cost solution. Stephen says the project had already spent too much on revisions and still lacked reliable PRs. Their later design keeps a stronger model for writing and moves mechanical work to cheaper reviewers. Cassandra describes this as counterintuitive because teams often give the planner, reviewer, or checker the strongest model. In their experience, a weaker writer turns feedback into a checklist even after several iterations, while claim checking is more mechanical.

### Relate assembles connected evidence before the model starts writing
[11:15](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=675s)
Stephen introduces Relate as an open source embedded graph database written in Rust. It stores relationships among the change, its rationale, code, checks, commands, and sources. The system builds a neighborhood brief containing connected facts and their attributions, then gives that packet to Tracetoon. Relate runs locally with compute and storage costs instead of LLM costs, and Stephen describes it as an in-memory database. The agent no longer has to repeatedly search session logs and reconstruct every fact during drafting. The evidence graph performs that preparation before the writer sees the material.

### A connected brief lets reviewers check specific claims against shared sources
[13:28](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=808s)
The neighborhood brief gives the writer and reviewers the same compact set of connected evidence. The example includes the working directory before a change-directory command, the exception where a command targets a repository or mentions a root, and the type check plus 71 tests run on the submitted content. Cheaper reviewers look for missing findings and compare claims with the draft. A stronger writer then produces the final description. This division reduces repeated searches and makes the review focus on whether each statement is supported by the collected evidence.

### PR Bench showed better judged descriptions and about half the token use for this project
[16:26](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=986s)
The speakers used PR Bench to compare agents across pull requests with audit traces, code, language-model judges, and human accuracy checks. They report quality improvements for both Claude and Codex. The examples shown move from 74 to 76 and from 82 to 86 on a 1-to-100 scale. They also report that the graph-based Tracetoon flow used about half as many tokens as the previous approach. Cassandra attributes the reduction to precomputing the evidence graph, making writing cheaper, and using less expensive reviewing. These results describe their Tracetoon evaluation rather than a guarantee for other projects.

### Provenance should remain outside the model conversation so uncertainty and support stay reviewable
[19:46](https://www.youtube.com/watch?v=gbeuC0ud1p4&t=1186s)
The speakers recommend preserving relationships and evidence in a separate project-specific graph layer. Agents are less effective at data analysis, evaluation, and curation when they must do all of that work inside the conversation. Relate can organize the sources first, then provide a connected brief to the writing and review stages. The useful artifact is the chain from a submitted change to its rationale, commands, timing, tested content, and final claim. The comic makes the explanation more approachable, but it does not replace the evidence behind the pull request.

## Notable quotes
- Cassandra Chin: "The what you submit is the final work, the program. But most of the work is the prompting and everything that came before and that doesn't get submitted with the program." (05:40)
- Cassandra Chin: "So he claims test passing and all the checks were run, and actually that's not what the code did." (10:00)
- Stephen Chin: "It's also the reviewing and the cross checking step which is more mechanical." (15:09)
- Stephen Chin: "Trace Tune with the graph solution takes about half as many tokens as it did previously." (18:19)

## Tools & references mentioned
- Tracetoon
- Relate
- Neo4j
- Show Tell
- PR Bench
- Claude
- Codex
- Opus
- Sonnet
- Haiku
- Luna
- AI Engineer Kids Day

## Who should watch
- You are using coding agents to open pull requests and need descriptions that connect claims to commands, code, and test evidence.
- Your review pipeline spends many tokens repeating searches through agent logs or rewriting descriptions that still contain unsupported statements.
- You are considering a local graph or evidence layer before generation and want an example of separating strong writing from cheaper claim checking.

## Editor's note

Stephen Chin says a passing-test claim is unreliable when recorded commands do not show that the submitted code was tested in the required sequence. Kitaru records every model call and tool result from a real agent run, then replays the run with the same inputs and tool responses. A failed production run can be replayed after a change to the model, prompt, or code.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Teach Your Agent to Prove It's Done](https://aietalks.com/talks/teach-your-agent-to-prove-its-done) (Nick Nisi & Zack Proser, WorkOS, 1:01:04)
- [Your Agent Fix Just Broke Something Else](https://aietalks.com/talks/your-agent-fix-just-broke-something-else) (Sofia Jakovcevic, Arize AI, 19:25)
- [All Code Is Slop (Except Mine)](https://aietalks.com/talks/all-code-is-slop-except-mine) (Lauren Tan, SpaceXSI, 17:32)
- [From Vibes to Production: Evaluating and Shipping AI Agents That Work 201](https://aietalks.com/talks/from-vibes-to-production-evaluating-and-shipping-ai-agents-that-work) (Laurie Voss, Arize AI, 42:17)
- [What Data from 20m Pull Requests Reveal About AI Transformation](https://aietalks.com/talks/what-data-from-20m-pull-requests-reveal-about-ai-transformation) (Nick Arcolano, Jellyfish, 17:57)
