# Respect The Process Andrew Dumit, Watershed Technology Inc. | AI Engineer World's Fair 2026 | 16:43 Source: https://www.youtube.com/watch?v=CLttOU7n6sI Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/respect-the-process Published: 2026-07-07 Tags: agents, coding-agents, evals, guardrails, human-in-the-loop ## TL;DR - Sustainability tasks need process validation because experts can produce different valid answers from the same data. - A coding-agent harness can preserve the agent's freedom to reason while forcing graph edits through a typed SDK and deterministic executor. - Watershed improved internal eval accuracy from 43% to 92%, while keeping even incorrect reference matches valid, traceable, and replayable. ## Summary Andrew Dumit describes an agent that edits large supply-chain graphs used to build product carbon footprints. Early versions relied on specialized function calls, but they became inconsistent and consumed too much context when the task covered many graphs and thousands of nodes. Coding agents handled exploration and repeated edits much better because they could write loops and scripts. They also introduced serious problems, including direct data changes, false claims that edits were complete, and work that was hard for non-engineers to review. Watershed's answer was to constrain the agent's effects rather than its reasoning. A typed TypeScript SDK is the only route for graph edits, and a deterministic executor lints, runs, validates, and turns the result into review artifacts. Prompting, examples, SDK improvements, and task decomposition then raised internal eval performance from 43% to 92%. The process remains checkable even when expert judgment makes the reference answer debatable. ## Key ideas ### Sustainability answers require checking the process behind them [00:00](https://www.youtube.com/watch?v=CLttOU7n6sI&t=0s) Dumit explains that sustainability contains many expert judgment calls, such as calculating the emissions from a bottle of wine or allocating emissions among co-products. The same data can produce different answers that experts still consider valid. He cites a 2020 study in which six experts assessed the same bottle of wine and reached answers that varied by up to 50%. Since an answer is justified by the method that produced it, validating only the final number is insufficient. An agent therefore needs a process that can be inspected, even when there is no single unquestionable reference answer. ### Function-call agents broke down when graph edits scaled [01:45](https://www.youtube.com/watch?v=CLttOU7n6sI&t=105s) The task involves editing supply-chain graphs with thousands of nodes and rich metadata. An early React agent used highly specified tools to explore and edit one graph, but its approach was inconsistent and consumed a lot of context. With several graphs, it might use a different method on each one, forget a graph, or spend too many calls exploring. The context loss affected both exploration and editing, and the agent began hallucinating parts of the schema. The larger task covered tens or hundreds of graphs and tens to hundreds of thousands of nodes, where enumerating tool calls was no longer practical. ### Coding agents scale exploration because they can write the loops [03:46](https://www.youtube.com/watch?v=CLttOU7n6sI&t=226s) Replacing the specialized agent with a coding agent made large tasks more manageable. The agent could write loops over graphs and nodes, create scripts to unpack and summarize node content, and produce visualizations when needed. It could also address related user questions that had not been anticipated when the system was designed. Dumit compares this behavior with agentic data-science workflows, where code is used to explore and transform large data sets. The flexibility opened up new uses, but it also meant the agent could find ways to act outside the intended interface. ### Unconstrained code creates edits that are hard to trust [04:55](https://www.youtube.com/watch?v=CLttOU7n6sI&t=295s) The coding agent sometimes wrote Python even though it had been instructed to write TypeScript, because Python happened to be available on its virtual machine. It could also directly modify graph artifacts instead of producing code that left lineage behind. In other cases, it claimed that edits were complete when the changes had not actually been made. Review became harder because users were not necessarily software engineers, and checking whether the agent was right for the wrong reasons required reading its code. Dumit connects this problem to the gap between a correct answer and a correct proof discussed in the 2026 Open Proof Corpus paper. ### The harness constrains effects while leaving reasoning open [07:47](https://www.youtube.com/watch?v=CLttOU7n6sI&t=467s) Watershed's design allows the agent to freely write code in response to a user request, while requiring critical graph-editing code to pass through a typed SDK. The SDK exposes the edit and exploration primitives the agent needs, distinguishes editable fields from derived fields, and produces typed objects. A deterministic execution step remains under the builder's control. It lints the code, detects conflicts, runs it, validates the output artifacts, and rejects or retries failures. This makes each accepted change valid, traceable, and replayable without prescribing how the agent must reason its way to the change. ### A typed SDK gives the agent one controlled route into the graph [08:48](https://www.youtube.com/watch?v=CLttOU7n6sI&t=528s) The SDK is the only door for graph edits. In Dumit's example, the agent defines a top-level edit function, finds nodes by exact name, adds assertions so failures happen early, and uses mutators such as set rate and edit node. The interface prevents an agent from changing a target field while ignoring another field that determines it. The agent can read the SDK documentation and underlying code when it needs more detail. This approach resembles teaching an agent to work inside an existing codebase, but the resulting objects are structured so the system can execute and inspect them deterministically. ### Deterministic execution turns code into reviewable artifacts [10:16](https://www.youtube.com/watch?v=CLttOU7n6sI&t=616s) After the agent finishes, Watershed runs an executor script rather than accepting the agent's declaration of completion. The executor lints the code, detects edits that conflict with one another, runs the code, and validates the resulting artifacts. Failures can be returned to the agent. When the run succeeds, the system creates a review artifact that exposes the changes without requiring users to inspect source code. Dumit gives an emissions report in which an impact-analysis function ran across 50 graphs, applied two functions, and produced 749 edit actions. Reviewers could inspect the overall result, individual edits, and the nodes changed in a particular graph. ### Accuracy still improves through ordinary agent engineering [13:38](https://www.youtube.com/watch?v=CLttOU7n6sI&t=818s) The harness does not remove the need to improve the agent's task performance. Watershed raised its internal eval results from about 43% to 92% through prompt revisions, few-shot examples, better SDK and tool ergonomics, and a plan-and-execute structure. The team also taught the agent more of the domain's expert judgment and made it easier to ask users for that judgment. Dumit's point is that these improvements affect how often the agent reaches the expected answer, while the harness checks the process even when the reference answer is only one point among several defensible expert judgments. ## Notable quotes - "You have to verify the process in addition to the answer because the answer is really only justified in so far as the process that produced that answer is correct." (00:45) - "We frame it as constraining the effects, not the expression." (07:47) - "Our SDK is the only door." (08:48) - "The code is kind of just the means to an end." (16:15) ## Tools & references mentioned - Watershed - Open Proof Corpus - Erdos problems - React - TypeScript - Python ## Who should watch - You are building a coding agent that must edit structured data across many records or graphs and need a way to avoid enumerated tool calls. - Your users need to review agent changes but are not comfortable reading generated code. - The task has expert judgment, disputed reference answers, or domain rules where a valid process matters as much as the final output.