Reliable agents need explicit plans, bounded tool outputs, production-based evals, and observability instead of prompts alone.
2
Production systems should put deterministic boundaries around model behavior, from OS-level file and network policies to structured routing and verification.
3
The talks argue that adaptive data, personal context, design judgment, and human review will matter as much as larger models as agents move into real work.
Summary
This day brings together practical lessons from building agents, deploying them, and designing around their limits. SallyAnn DeLucia describes Arize AI's Alex and its use of explicit to-do states, bounded JSON abstractions, production traces, and agent-assisted debugging. Other speakers address synthetic data, code-mode tool calling, harnesses, document parsing failures, OS-level guardrails, and agents that manage their own compute. The afternoon turns to robotics, creative tools, company knowledge, and long-running coding systems. Speakers repeatedly return to the same engineering problem: models are capable, but their behavior becomes unreliable when context, permissions, evaluation, and execution are left implicit. The proposed responses vary from deterministic boundaries and specialized encoders to adaptive data and richer human context. The event also makes a case for building in Singapore, with Agrim Singh describing how the conference was organized around builders, workshops, access to speakers, and scholarships for students.
Explicit planning keeps agents from forgetting unfinished work
SallyAnn DeLucia says agents often complete the first item in a request and lose the second or third because of an attention problem. Alex addresses this with explicit planning tools and states. Its tools write, update, and read to-dos, while tasks can be pending, in progress, completed, or blocked. The plan is injected outside conversation history so truncation cannot remove it. A finish gate rejects completion until all required tasks are done, except when a task is blocked on human input. Her conclusion is that code-enforced state, tool calls, and examples of good planning work better than abstract instructions.
Context management means preserving structure while compressing values
Alex handles large platform outputs with a large JSON abstraction. Instead of truncating a response to its first tokens, it keeps fields and array structure while shortening large string values. The agent receives an identifier and can retrieve more data when needed. De Lucia also describes small composable tools, including jq and a regular-expression search tool, which let the agent slice and aggregate data without adding one enormous tool. Every tool output has a hard token budget, with a 10,000-token limit in their system, so overflow becomes another turn rather than an unpredictable context failure. She also warns that tool responses may contain customer data and require careful log handling.
Production traces provide better evaluation cases than handwritten golden answers
De Lucia says manual spreadsheet-based testing and vibe checking did not scale as Alex changed. The team instead treats production traces as ground truth and turns real user sessions into test cases. They use decision-point tests for individual components and trajectory tests that replay production choices step by step. Since language-model outputs can differ while expressing the same correct result, checks should match facts rather than exact phrasing. An evaluator model judges semantic success, with prompts that define success for each step. These tests run ad hoc, in CI, and in Arize visualizations so the team can track performance over time.
Production agents need deterministic boundaries around nondeterministic models
JJ Geewax argues that production systems should stop treating one language model as a single router for every problem. A model can classify a request, convert it into structured data, transform one known structure into another, and generate a response. Routing can be an LLM decision, but the surrounding steps should be constrained and understandable. Safety checks can use a context-free model call or a smaller classifier. For multimodal systems, he combines fast local models that process many frames with slower, more capable models that inspect selected frames. His point is practical: use models for the difficult flexible parts and deterministic systems for decisions where an unexpected answer would be unacceptable.
A harness improves reliability by controlling the environment around an agent
Tejas Kumar defines an agent harness as everything around the agent that gives it a better chance of reliable execution. Its components include tools, a language model, context management, guardrails, an agent loop, and verification. In his browser demo, a weak model falsely claims it upvoted a Hacker News story after reaching a login screen. The harness adds iteration and message limits, trims context, verifies whether the upvote actually happened, detects an unrecovered login redirect, and retries. It then handles login outside the agent's tool loop. The result works without changing the prompt or upgrading the old model. He presents this as a way to do more with less.
OS-level policies avoid the failure pattern created by permission prompts
Jun Yu Tan compares coding-agent permission fatigue with SQL injection. Repeated approval prompts interrupt work, so users often disable them, leaving agents with broad access to files, credentials, and environment secrets. Tusk's Fence moves enforcement below the agent. It restricts file paths, filters network access through allowed domains, and checks commands, including nested shells. A policy file defines what the process can see, reach, or run. Tan places this between probabilistic action classification and stronger isolation such as containers or microVMs. The aim is to change what the agent can do rather than repeatedly asking the agent to behave.
Adaptive systems should optimize data and compute instead of only growing model size
Sara Hooker argues that the industry has treated larger models as the main route to progress, but the returns from brute-force scaling are weakening. She points to smaller models outperforming larger ones, redundant weights, weight removal, and the value of high-quality data. She describes a future built around adaptive compute, interaction with the world, and continuous learning. Adaption Labs is working on adaptive data and AutoScientists, which automatically learns how to optimize data and models for a task. Her broader claim is that intelligence should adapt across the stack, from data through the interface, rather than forcing everyone to use the same static model.
Real-world robotics needs useful deployment data and intervention, not only lab performance
Daniel and Siddharth Krishnan describe teleoperated robots working in insect farms, laundries, food preparation, and hospitality. Teleoperation is expensive and difficult to scale, but it produces data that matches the robot's hardware, environment, and task. Their proposed loop starts with useful commercial work, uses the resulting data for supervised fine-tuning, and then adds tele-supervision when the robot reaches an unreliable plateau. They call 80% autonomy insufficient for production because one failure in five can directly affect a customer. Remote, low-latency supervision can let one operator support multiple robots while corrections become new training data.
Creative AI needs human taste, iteration, and editable structure
Josh Newton says AI behaves like a magic pencil and should assist creative judgment rather than replace it. He argues that models tend toward common patterns because they are trained on existing work, so speed can push products toward average quality. His examples include Orbit, a subscription-tracking app shaped by craft and a specific audience, and custom prototyping tools that let him explore states and edge cases quickly. He recommends lateral inspiration, such as starting from a poster rather than another app, and repeated iteration instead of accepting the first prompt. Alex Lee and Priyaa Kalyanaraman extend this argument to design systems and layered graphics, where code, brand rules, editable layers, and specialized models give agents more useful context.
"Planning is the way for your agent to first decide what it is it needs to do before actually actioning on it."12:24
Who should watch
You are building an agent that loses tasks, overruns its context, or claims success without verifying the result.
Your team is moving an agent from a demo into production and needs concrete choices around evaluation, permissions, fallbacks, and observability.
You work on robotics, design tools, company knowledge, or coding infrastructure and want examples of how model behavior can be constrained by better systems.
