Agents need production evaluations that measure reliability, cost, and task-specific performance, rather than relying on one static benchmark.
2
The simplest useful agent is usually a model operating in a loop with tools, an environment, and a clear system prompt.
3
Production systems improve when teams constrain agent scope, add verification and fallbacks, and let models use more compute where it helps.
Summary
This day-long track examines how teams are building agents that do real work. The talks cover research agents, coding tools, financial workflows, voice calls, education, and personal assistants. A repeated message is that agent engineering is a systems discipline. Models remain stochastic, so teams need strong evaluations, guard rails, human review, fallbacks, and ways to inspect failures. Anthropic recommends starting with workflows when a task can be mapped explicitly, then keeping agents to a simple tool-using loop. Other speakers describe fine-tuning smaller models for cost and latency, using reinforcement learning to teach skills, and designing products around asynchronous work and user trust. The later talks look further ahead, toward agents that co-create with people, help children learn by making things, and run locally with access to deeply personal context. The speakers are optimistic about capability growth, while being direct about the engineering work still needed for dependable deployment.
AI engineering is becoming its own discipline around agents
swyx frames the track as a move from general AI engineering toward agent engineering. He says the field is still early and is often misunderstood from both sides: machine-learning engineers may see it as prompts around a model, while software engineers may see it as ordinary software calling an LLM API. He argues that agent systems combine models with tools, control flow, delegated authority, long-running work, and multi-step completion. He also points to falling model costs, better reasoning and tool use, greater model diversity, reinforcement-learning options, and faster inference as reasons agents are becoming practical now. He is skeptical of declaring every year the year of agents, but says coding, support, and deep research already show strong product-market fit.
Agents need reliability-focused evaluation instead of impressive demos
Shashank Kapoors argues that agent evaluation must be treated as part of the engineering toolkit. He cites failures in legal products, scientific research agents, and CUDA-kernel optimization where public claims did not survive closer testing. Agent evaluations differ from language-model tests because agents act in environments, can run for open-ended lengths, incur variable costs, and are often built for a specific task. His Princeton work combines accuracy and cost on a Pareto frontier, while the Berkeley work he cites adds domain experts who revise evaluation criteria. He also separates capability from reliability. An agent may produce a correct result sometimes while failing too often for a consequential product. Verifiers can fail too, so system design has to account for stochastic model behavior.
Deep research agents need asynchronous state, iterative planning, and context management
Google's Mukun Sudar and Arush describe Gemini Deep Research as a product that trades latency and compute for a more complete answer. The user first sees an editable research plan, then can watch the system browse, and finally receives a long report with sources and an interactive artifact for follow-up questions. Behind the product, the system must recover from failed services, preserve state across devices, decide which subproblems can run in parallel, and plan from partial information. The speakers show examples where the agent must find missing scholarship standards, combine pricing and certification information from separate pages, resolve ambiguous entities, and handle inconsistent web layouts. They also describe selectively compressing older research into notes and retrieval so the context remains usable across follow-up tasks.
Use agents for complex, valuable work and keep their core loop simple
Barry Zhang says agents should scale complex and valuable tasks, rather than replace every workflow. If a decision tree can be written down, an explicit workflow is usually cheaper and easier to control. Agents become more attractive when the task is ambiguous, the output has high value, and errors can be found or constrained. His basic architecture has an environment, tools that provide actions and feedback, and a system prompt that defines goals and limits. The model runs in a loop. He recommends building these parts before adding elaborate orchestration. Developers should also inspect the agent's limited context, which may contain only 10,000 to 20,000 tokens about the current state. For computer-use systems, this means reasoning from a screenshot and tool result, with no guarantee that an action worked.
Production agents improve through an agent development life cycle
Zach Renau presents Sierra's process for improving customer-service agents. He treats each agent as a product that needs engineering, product management, quality assurance, and operations. A production conversation can reveal a missing API call, a timeout, or an opportunity to go further for the customer. That observation becomes an issue, then a test, and eventually part of a new release. An agent can grow from a small set of tests to hundreds or thousands. Sierra applies this process across chat and voice, using the same underlying agent while adapting to each channel. Renau compares language models with traditional software: they are flexible and creative, but also slower, more expensive, and nondeterministic. The life cycle combines model capabilities with conventional software where deterministic behavior is needed.
Reinforcement learning can teach agents skills through environments and feedback
Will Brown describes reinforcement learning as a possible next step for agents that have reached the limits of prompt tuning. In this setup, a model acts in an environment, receives a score, and learns to repeat strategies that work. He connects this approach to reasoning models such as OpenAI's o1 and DeepSeek's R1, where longer chains of thought can emerge from training rather than being manually written. For agents, the open problems include building useful environments, defining rewards, handling tool calls, controlling cost, and avoiding reward hacking. Brown's example of a small model trained on math tasks attracted attention because the code exposed the rubric and invited modification. He calls this approach rubric engineering, where the scoring rules shape the behavior the model learns.
Smaller fine-tuned models can make high-volume agents affordable
Mustafa Ali and Kyle Corbett explain how Method Financial moved an unstructured-data workflow from a costly GPT-4 system toward a smaller custom model. Method needed to extract liability data from bank conversations and other sources, with high request volume, low latency, and low error rates. Prompt engineering helped at first, but the prompts became long and fragile, and the initial production bill reached $70,000 in one month. The team measured quality, latency, and cost on real production tasks, then fine-tuned an eight-billion-parameter Llama 3.1 model using data generated during production. The resulting model met the quality threshold while reducing latency and cost. Their advice is to benchmark hosted models first, then fine-tune when prompting cannot meet the business requirements. They did not need to buy GPUs.
Agent products need a narrow scope, good interfaces, and protection around actions
Nick Kotakis says reliable voice agents come from the conversation and the surrounding workflow, rather than from an unusually realistic voice. SuperDial makes phone calls for healthcare administration and returns structured answers, with human fallback when the bot cannot finish. The team treats boring completed calls as success and audits calls to improve office hours and phone-tree navigation. Voice adds transcription, speech generation, audio review, real-time latency, pronunciation, spelling, and end-to-end testing. Kotakis recommends tracking time to first byte, keeping fallbacks for every major service, and testing with simulated phone trees and audio files. He also argues that vertical products should focus on the last mile, including domain integrations and conversational content, because those are what make a generic voice system useful.
Future agents will collaborate with people and may need to run locally
The final talks extend agent design beyond today's task automation. Karina Nguyen describes a progression from next-token pretraining, to post-training, to reinforcement learning over chains of thought and tool use. She expects agents to become co-innovators through shared canvases, asynchronous tasks, human feedback, and multimodal collaboration. Stefania Druga shows how children can learn AI by training models and programming robots in Scratch-based environments. Her studies found that young people wanted help with ideas, explanations, and motivation while keeping control of their projects. Soumith Chintala argues that personal agents need broad context and the ability to act, which creates privacy and trust concerns. He prefers local systems because a personal agent may access messages, calendars, finances, and private thoughts. He sees the Mac Mini as a practical current device, while noting that local multimodal models and safeguards still need work.
"The challenge for AI Engineers is to figure out what sorts of software optimizations and abstractions are needed for working with inherently stochastic components like LLMs."44:02
Who should watch
You are deciding whether a task should use a workflow, a single agent, or several agents, and need practical criteria for making that choice.
You are taking an agent from a demo into production and need to plan evaluations, cost controls, fallbacks, human review, and failure recovery.
You are building research, coding, voice, finance, education, or personal-assistant products and want examples of the design constraints in each setting.
