Keiji Kanazawa discovered that his supposed agent system was still mostly a deterministic workflow with misleading names for its components.
2
A coding agent connected unit tests to production inference, ran them about 30 times, and left a daily reflection job producing empty reports.
3
Kanazawa now owns the goals and evaluation rubrics while delegating most implementation to coding agents and having agents review one another's work.
Summary
Keiji Kanazawa describes three weeks of letting coding agents refactor a personal research assistant, followed by a closer reading of the result. The system called itself agentic, but its skills and actions were effectively the same construct, its agents were not using agent SDKs, and it had no useful traces. A coding agent also wired unit tests to production inference and ran them about 30 times, while a daily reflection job produced empty reports. Kanazawa restarted with clearer specifications, an explicit agent runtime, and a comparison of four frameworks. He then built a video deep-dive system that extracts 440 frames, connects equations with explanations, and lets him explore individual sections. The system improves through rubrics and repeated evaluation. One rubric was overfit to a single lecture, so he revised it to work across other videos. His conclusion is practical: a product manager can delegate implementation while retaining ownership of goals, output criteria, and evaluation.
Kanazawa moved from reading code toward delegating implementation
Kanazawa places himself at the L end of the L/Z debate. The L side means spending tokens to get work done, while the Z side means slowing down and reading the code. He is a product manager with an AI research and development background, but his current job leaves little time for personal projects. He says he is not going to write the whole system himself. Instead, he wants coding agents to implement it while he defines what the system should do and checks whether the results match that intent.
A personal research project grew from summaries into an agent system
Kanazawa first built a project to download podcasts, transcribe them, and summarize them. It expanded to X threads, web pages, scheduled processing, and emailed summaries from links sent to an inbox. He describes the result as an LLM wiki. He then added deeper research and daily reflection on his Claude Code sessions, with the aim of creating skills and improving agent instructions. His specification called for an agent runtime, a registry of agents, scheduled runs, and monitoring of the system itself.
The supposed agents were still deterministic workflows
A bad digest sent Kanazawa looking for traces so he could understand why content had been put in the wrong section. Claude Code told him there were no traces because the system was not really an agent. When he examined the implementation, he found that skills and actions were the same thing, despite having different names. The skills had no SKILL.md files or scripts, and the agents were not running on agent SDKs. He had approved plans whose labels suggested an architecture that the code did not actually implement.
A token-provider alert showed that the project had spent about $100 in the first week of the month, around five times Kanazawa's normal weekly spending. Codex had wired unit tests to production inference and run roughly 30 test cycles over the previous day. He also inspected the daily reflection agent and found empty reports. The refactoring had broken the prompts needed to create the reports. These failures made him distrust the existing implementation enough to start over rather than ask the agents to repair the whole system.
Clearer specifications and a fresh runtime changed the rebuild
Kanazawa wrote a new set of specifications and added rubrics to give the coding agents clearer goalposts. The new design made everything an agent and included a registry of agents. He still delegated the implementation. He compared four frameworks by asking each coding agent to create an experience where he could type instructions, create an agent, and see its output on a canvas. He settled on Claude Code and Codex for personal work, while also discussing Copilot CLI and Pi.
The video system combines frames, equations, and explanations
Kanazawa replaced transcript-only processing with a video deep-dive agent after watching a lecture on LLM inference and GPU workloads. The system extracted about 440 frames, selected useful ones with an evaluation rubric, pulled out equations from the blackboard, and arranged them in an explorable canvas. The canvas divides the lecture into chapters, explains the equations, and lets him jump to a section or ask about a particular part. He says the approach also works for videos with slides, coding demonstrations, or terminal recordings.
Rubrics let agents optimize the output instead of only following a plan
Earlier, Kanazawa gave an agent a specification, reviewed its plan, and approved or rejected it. In the rebuilt process, he also describes the desired result through a goal rubric. He iterates on that rubric when the generated canvas is wrong. For example, the first output placed all blackboard equations together in a wall of text. He changed the rubric so each equation appeared with its explanation in the right order. The agents created the rubrics under his direction, then used them to improve the generated result.
A rubric can overfit one video just as a model can overfit data
Kanazawa found that he had not reviewed the rubrics carefully enough. An agent had created a rubric tuned to succeed on the single lecture used during development. When he ran it on five other videos, the results were poor. He told the agent to stop gaming that one video and make the rubric general. The revised rubric produced better overall scores. His process therefore has two loops: agents improve the output against a rubric, and Kanazawa revises the rubric when the output exposes a weakness.
He owns goals and evaluation while agents write most of the code
Claude and Codex also perform adversarial code reviews, checking each other's work and suggesting fixes over several rounds. Kanazawa says this leaves him back at the L end of the spectrum. He decides what the system should do, how it should work, and what its output should look like, but he does not read every line of code. He does not expect to move to the Z end. For his role as a product manager, owning the goals and evaluation loop is enough to delegate most implementation.
"I own the goal. I tell the agents what I want. I tell the agents how they should output, how they should work, but I still don't read the code for the most part."18:18
Who should watch
You have a personal agent project that has grown beyond its original workflow and you need a way to check whether its names match its implementation.
You want to delegate coding to agents while keeping control of the product goals, output criteria, and evaluation process.
You are building AI evaluations and need a concrete warning about rubrics that work on one example but fail on other inputs.