Your PR Says the Tests Passed. Prove It.

Stephen Chin, Neo4j, Cassandra Chin, Tracetoon creator23:01 · Nov 2026 · 451 views
Thumbnail for Your PR Says the Tests Passed. Prove It. Watch on YouTube
TL;DR
  1. 1

    Tracetoon reconstructs agent activity across sessions so pull requests can explain what changed, why it changed, and what evidence supports the claims.

  2. 2

    A passing-test claim is unreliable when the recorded commands do not show that the submitted code was tested in the required sequence.

  3. 3

    Relate builds a local evidence graph before drafting, allowing a stronger writer to use a compact brief while cheaper reviewers check claims and reduce token use.

Summary

Stephen Chin and Cassandra Chin examine why AI-generated pull requests can contain fluent but unsupported claims. Tracetoon gathers context from agent sessions, code changes, commands, and audit trails, then uses that material to write and review a pull request. Their example exposes a serious error: a PR claimed that tests passed even though the trace showed incompatible commands running at the same time. Repeated revisions improved the prose but consumed tokens and still allowed mistakes. The newer design uses Relate, an embedded Rust graph database, to connect a change with its rationale, command history, timing, and tested code. This produces a connected evidence brief before writing begins. A higher-quality model writes the PR, while less expensive reviewers check claims against the brief. Their PR Bench comparisons showed better judged descriptions and roughly half the token usage for this project. The speakers present this as an engineering pattern, not a guarantee for every codebase.

Key ideas
00:00

Tracetoon began as a way to recover the work hidden behind AI-generated code

Cassandra Chin built Tracetoon after seeing pull requests rejected because their descriptions contained incorrect information or were hard to read. The submitted program shows the final artifact, while the prompting and agent activity that produced it usually remain outside the repository. Her earlier Show Tell work focused on tracing how students created programs. Tracetoon applies that same idea to pull requests by collecting agent context, code information, and session history. It can also add a comic that explains the change, although the engineering purpose is to improve the accuracy and readability of the PR.

04:20

Reading one agent session is insufficient because context becomes compressed and inaccurate

A coding agent often writes a pull request by reading its own session log. Stephen Chin says the agent's context window gets compressed as the session continues, so the resulting description can become less accurate. Tracetoon instead searches multiple sessions on the developer's computer to reconstruct what happened across the branch. That evidence can include the working directory before a command changes directories, the reason for a command, and information about possible test-data leakage between repositories. This gives the PR more than a line-by-line explanation of the diff.

07:24

Repeated research and revision can improve a PR while still leaving factual mistakes

The original Tracetoon flow finds traces for a branch, researches them, reviews the draft, adds context, and repeats the process until it has enough evidence. Cassandra says it produced better descriptions and fewer inaccuracies than working without Tracetoon, both subjectively and through PR Bench. It still sometimes made mistakes during later revisions, when the system tried to rewrite with more context. One example claimed that no leader passing run was recorded, even though the trace showed bun test running at the same time as bun x tsc. Those commands had to run sequentially for the test to pass, so the claim did not match the actual execution.

10:23

Using cheaper writers reduced cost but also reduced the quality of the pull request

The speakers tried replacing expensive writing models with cheaper ones to address token spending. The result was worse quality rather than a useful cost solution. Stephen says the project had already spent too much on revisions and still lacked reliable PRs. Their later design keeps a stronger model for writing and moves mechanical work to cheaper reviewers. Cassandra describes this as counterintuitive because teams often give the planner, reviewer, or checker the strongest model. In their experience, a weaker writer turns feedback into a checklist even after several iterations, while claim checking is more mechanical.

11:15

Relate assembles connected evidence before the model starts writing

Stephen introduces Relate as an open source embedded graph database written in Rust. It stores relationships among the change, its rationale, code, checks, commands, and sources. The system builds a neighborhood brief containing connected facts and their attributions, then gives that packet to Tracetoon. Relate runs locally with compute and storage costs instead of LLM costs, and Stephen describes it as an in-memory database. The agent no longer has to repeatedly search session logs and reconstruct every fact during drafting. The evidence graph performs that preparation before the writer sees the material.

13:28

A connected brief lets reviewers check specific claims against shared sources

The neighborhood brief gives the writer and reviewers the same compact set of connected evidence. The example includes the working directory before a change-directory command, the exception where a command targets a repository or mentions a root, and the type check plus 71 tests run on the submitted content. Cheaper reviewers look for missing findings and compare claims with the draft. A stronger writer then produces the final description. This division reduces repeated searches and makes the review focus on whether each statement is supported by the collected evidence.

16:26

PR Bench showed better judged descriptions and about half the token use for this project

The speakers used PR Bench to compare agents across pull requests with audit traces, code, language-model judges, and human accuracy checks. They report quality improvements for both Claude and Codex. The examples shown move from 74 to 76 and from 82 to 86 on a 1-to-100 scale. They also report that the graph-based Tracetoon flow used about half as many tokens as the previous approach. Cassandra attributes the reduction to precomputing the evidence graph, making writing cheaper, and using less expensive reviewing. These results describe their Tracetoon evaluation rather than a guarantee for other projects.

19:46

Provenance should remain outside the model conversation so uncertainty and support stay reviewable

The speakers recommend preserving relationships and evidence in a separate project-specific graph layer. Agents are less effective at data analysis, evaluation, and curation when they must do all of that work inside the conversation. Relate can organize the sources first, then provide a connected brief to the writing and review stages. The useful artifact is the chain from a submitted change to its rationale, commands, timing, tested content, and final claim. The comic makes the explanation more approachable, but it does not replace the evidence behind the pull request.

"The what you submit is the final work, the program. But most of the work is the prompting and everything that came before and that doesn't get submitted with the program."Cassandra Chin05:40
Who should watch
  • You are using coding agents to open pull requests and need descriptions that connect claims to commands, code, and test evidence.
  • Your review pipeline spends many tokens repeating searches through agent logs or rewriting descriptions that still contain unsupported statements.
  • You are considering a local graph or evidence layer before generation and want an example of separating strong writing from cheaper claim checking.