Frontier coding models are making code generation cheap, so engineering effort is moving toward specifying work, designing systems, verifying outcomes, and accepting responsibility.
2
Autoresearch works best when agents have a measurable objective, a controlled search surface, useful feedback, and enough compute to run repeated experiments without reward hacking.
3
Long-running agents need explicit memory, traces, scenario-based evaluation, and human judgment because success depends on behavior across workflows rather than one model response.
Summary
This Day 2 program connects several views of what comes after basic coding agents. Anthropic presents Fable as a model whose capabilities depend heavily on its harness, tools, context, and the unknowns engineers expose or hide. Sonar, Meta, and Arize argue that agent systems need verification built into their loops, with production traces, scenario tests, human review, and layered checks. Speakers from Google DeepMind, Prime, Weights & Biases, and others describe autoresearch systems that search over code, prompts, kernels, retrieval pipelines, and agent policies. Their results also expose limits: agents often optimize the measured objective without discovering genuinely new ideas. Other talks focus on memory, cross-repository state, shared context, visual artifacts, mobile development, and multi-agent social behavior. The recurring engineering lesson is that models are only one component. The surrounding specifications, evaluation design, memory, tools, and human decisions determine whether the system produces useful work.
Fable becomes more capable when engineers remove constraints from the harness
Tariq Shihipar describes Fable as a new class of model whose capabilities appear in "spiky" ways. A chat model cannot reliably find the two Pokémon whose names end in "aw," but Claude Code can fetch the full list and write a filtering script. Shihipar calls this capability overhang. The same pattern appears in coding, where tools such as bash let a model build and search its own context instead of receiving an entire codebase in one prompt. He says Anthropic removed 80% of Claude Code's system prompt because examples and prohibitions began to constrain a more capable model. Engineers need to discover what the model can do with the right environment.
Engineers can use models to uncover the unknowns in their own specifications
Shihipar separates a project into known knowns, known unknowns, unknown knowns, and unknown unknowns. Fable makes this more important because it can travel through a larger part of the problem and encounter decisions that the original prompt never specified. He recommends a "blind spot pass" over code, Git diffs, Slack, or other sources; asking the model to interview the engineer; creating several radically different prototypes; and supplying reference implementations as another map of the desired result. He also suggests logging deviations and quizzing yourself about what the model changed. The aim is to keep the human in the loop without forcing the human to write every implementation detail.
Verification has to surround agentic coding instead of being added after generation
Tariq Shakat argues that coding agents can complete long tasks while still being wrong. A benchmark may show a task completed at a 50% success rate, while an enterprise may need reliability in the nines. Sonar's proposed agent-centric development cycle puts guide, verify, and solve around code generation. Guidance includes both context, such as repository structure, and constraints, such as allowed dependencies and intended architecture. Verification combines algorithmic checks with agentic checks for intent and business logic. Shakat says customers using layered verification reported 44% fewer AI-derived production outages, while one bank saw a 92% reduction in issues with the full approach.
Agents need shared perception and recovery to complete work across application seams
The Amazon AGI presentation distinguishes computer-use capabilities from completing real work. An onboarding process may require account setup, Slack access, meetings, and equipment across several systems. Agents can perform individual clicks, but they often fail between the steps because they do not observe results or recover from failure. The proposed perception agent follows a robotics-like loop of perceiving, planning, and acting. It reads the rendered screen, sees the same pixels as the user, checks its own output, and can accept direct annotations such as pointing to a heading that should change. The demonstrated harness includes annotation and verification, including visual checks and automated user-flow tests.
Autoresearch can improve systems when the objective and search space are tightly controlled
Several talks describe autoresearch as a loop in which an agent proposes a change, runs an experiment, measures the result, and keeps or reverts the change. Google DeepMind's Benois Schillings says code is unusually suited to this because it has abundant training data and can be compiled and tested. Prime's experiments on optimizer speedruns show that agents can beat human records, but they mostly combine known ideas rather than inventing new mechanisms. The proposed remedy is a multi-agent system with generators, judges, scaling tests, and human guidance. The quality of the evaluator and the abstraction around the code determine what the search can discover.
Production evaluation measures system behavior across workflows, not just model answers
Meta's Nishan Gupta says benchmarks measure model capability while production measures system behavior. Agent evaluation must cover planning, tool use, execution, recovery, memory, safety, and coordination. He recommends scenario-based offline tests for workflows such as support, coding, and research, followed by production telemetry from traces, user outcomes, escalations, failures, and feedback. Evaluation becomes a continuous service after deployment because models, prompts, tools, and user behavior change. Arize's Aparna Dinakaran makes a related case for agent-as-a-judge systems that inspect varied trajectories and find patterns such as repeated tool calls or inefficient loops that fixed LLM-as-a-judge rubrics can miss.
Long-horizon agents need structured memory and provenance, not just more retrieval
Stefania Dug presents memory as a write, manage, and read control loop around a model. Her harness keeps a trace core, tests several recall policies, and stores an archival record across sessions. On long-horizon tasks where the relevant answer falls outside the active context, a ranked decision ledger performed better than no memory or a simple gate deciding whether memory was needed. She also found that bad memory can cost more tokens and send an agent in the wrong direction. The multi-agent game Project Paradox reaches a similar conclusion: agents must preserve who said something, whether it was firsthand or uncertain, and whether a belief has changed. Retrieval alone does not preserve social consistency.
Autoresearch changes the value of prompts and code abstractions
Aiden's parameter-golf results are presented as evidence that an autonomous system can run experiments continuously and publish work that other engineers reuse. The system's strongest contributions came from finding ideas in papers and community submissions, combining them, and implementing them under competition constraints. The speaker compares an evaluation to a loss function and a codebase abstraction to a model architecture. A loose interface can let an agent exploit data leakage or another shortcut, while a stricter interface can prevent that behavior. Reflective optimization makes the same argument for prompts and agent harnesses: textual feedback can produce large behavior changes with few examples, but candidate selection must preserve alternatives instead of getting trapped in a local optimum.
Human work is moving toward judgment, specification, and accountability
The closing talks argue that faster generation does not remove engineering. Bash says teams must decide what is worth building because most ideas in an internal hackathon produced no business value. Addy Osmani frames the human role around evidence and verdicts: someone must decide whether work ships, what risk is accepted, and who owns the result. He warns about cognitive debt, borrowed confidence, and orchestration tax when agents run faster than people can understand them. Other speakers make the same point through different systems: Polygraph gives agents cross-repository context and memory, Nori uses HTML as a structured medium for visual artifacts, and mobile sandboxes let designers and QA work directly against running builds. The common boundary is evidence and responsibility.
"The question is no longer did the model generate the right answer. The question is did the system behave correctly."3:36:05
Who should watch
You are building coding or research agents and need a practical way to define experiments, constrain changes, and decide what counts as an improvement.
Your agent works across long workflows, repositories, tools, or users, and failures happen between steps rather than in a single response.
You are deciding what engineers should own as generation becomes cheaper, especially evaluation, system design, product judgment, and production risk.
