# AI Engineer Singapore Day 2

Kaspar Hidayat, 65 Labs & SallyAnn DeLucia, Arize AI & Timothy Lin, Resaro & Abhishek Kankani, Cloudflare & Tejas Kumar, IBM & JJ Geewax, Google DeepMind & Geoff Huntley, Independent & Vincent Koc, OpenClaw Foundation & Vishnu (Vish) Hari, Ego AI & Ben Guo, Zo Computer & Matthias Lubken, Tavon AI & Josh Newton, Microsoft AI & Sam Bhagwat, Mastra & Pierre-Loic Doulcet, LlamaIndex & Jun Yu Tan, Tusk & Sara Hooker, Adaption Labs & Vincent Wu, MiniMax & Daniel Krishnan & Siddharth Krishnan, The Robot Company & Justin Baird, Tesseract & Kai Ming & Aravind (SK) Kandiah, Bifrost & Julia Kim, OpenGraph Labs & Suveen Ellawela, Cortex AI & Keziah & Jay Demetillo & Alex Lee, Magic Patterns & Sabina Cabrera, MagicPath & Priyaa Kalyanaraman, Lica World & Conor Brennan-Burke, Hyperspell & Heng Hong Lee, Lightsprint & Louis Knight-Webb, Vibe Kanban & Harsha Khurdula, Interfaze AI & Hrishi Olickel, Southbridge & Henry Mao, Smithery & Rach Pradhan, Independent & Agrim Singh, AI Engineer | AI Engineer Europe 2026 | 9:28:00

Source: https://www.youtube.com/watch?v=m12vGjfbNlo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-engineer-singapore-day-2
Published: 2026-05-17
Tags: evals, guardrails, observability, reliability

## TL;DR
- Reliable agents need explicit plans, bounded tool outputs, production-based evals, and observability instead of prompts alone.
- Production systems should put deterministic boundaries around model behavior, from OS-level file and network policies to structured routing and verification.
- The talks argue that adaptive data, personal context, design judgment, and human review will matter as much as larger models as agents move into real work.

## Summary
This day brings together practical lessons from building agents, deploying them, and designing around their limits. SallyAnn DeLucia describes Arize AI's Alex and its use of explicit to-do states, bounded JSON abstractions, production traces, and agent-assisted debugging. Other speakers address synthetic data, code-mode tool calling, harnesses, document parsing failures, OS-level guardrails, and agents that manage their own compute. The afternoon turns to robotics, creative tools, company knowledge, and long-running coding systems. Speakers repeatedly return to the same engineering problem: models are capable, but their behavior becomes unreliable when context, permissions, evaluation, and execution are left implicit. The proposed responses vary from deterministic boundaries and specialized encoders to adaptive data and richer human context. The event also makes a case for building in Singapore, with Agrim Singh describing how the conference was organized around builders, workshops, access to speakers, and scholarships for students.

## Key ideas
### Explicit planning keeps agents from forgetting unfinished work
[10:47](https://www.youtube.com/watch?v=m12vGjfbNlo&t=647s)
SallyAnn DeLucia says agents often complete the first item in a request and lose the second or third because of an attention problem. Alex addresses this with explicit planning tools and states. Its tools write, update, and read to-dos, while tasks can be pending, in progress, completed, or blocked. The plan is injected outside conversation history so truncation cannot remove it. A finish gate rejects completion until all required tasks are done, except when a task is blocked on human input. Her conclusion is that code-enforced state, tool calls, and examples of good planning work better than abstract instructions.

### Context management means preserving structure while compressing values
[15:42](https://www.youtube.com/watch?v=m12vGjfbNlo&t=942s)
Alex handles large platform outputs with a large JSON abstraction. Instead of truncating a response to its first tokens, it keeps fields and array structure while shortening large string values. The agent receives an identifier and can retrieve more data when needed. De Lucia also describes small composable tools, including jq and a regular-expression search tool, which let the agent slice and aggregate data without adding one enormous tool. Every tool output has a hard token budget, with a 10,000-token limit in their system, so overflow becomes another turn rather than an unpredictable context failure. She also warns that tool responses may contain customer data and require careful log handling.

### Production traces provide better evaluation cases than handwritten golden answers
[18:55](https://www.youtube.com/watch?v=m12vGjfbNlo&t=1135s)
De Lucia says manual spreadsheet-based testing and vibe checking did not scale as Alex changed. The team instead treats production traces as ground truth and turns real user sessions into test cases. They use decision-point tests for individual components and trajectory tests that replay production choices step by step. Since language-model outputs can differ while expressing the same correct result, checks should match facts rather than exact phrasing. An evaluator model judges semantic success, with prompts that define success for each step. These tests run ad hoc, in CI, and in Arize visualizations so the team can track performance over time.

### Production agents need deterministic boundaries around nondeterministic models
[1:27:13](https://www.youtube.com/watch?v=m12vGjfbNlo&t=5233s)
JJ Geewax argues that production systems should stop treating one language model as a single router for every problem. A model can classify a request, convert it into structured data, transform one known structure into another, and generate a response. Routing can be an LLM decision, but the surrounding steps should be constrained and understandable. Safety checks can use a context-free model call or a smaller classifier. For multimodal systems, he combines fast local models that process many frames with slower, more capable models that inspect selected frames. His point is practical: use models for the difficult flexible parts and deterministic systems for decisions where an unexpected answer would be unacceptable.

### A harness improves reliability by controlling the environment around an agent
[51:07](https://www.youtube.com/watch?v=m12vGjfbNlo&t=3067s)
Tejas Kumar defines an agent harness as everything around the agent that gives it a better chance of reliable execution. Its components include tools, a language model, context management, guardrails, an agent loop, and verification. In his browser demo, a weak model falsely claims it upvoted a Hacker News story after reaching a login screen. The harness adds iteration and message limits, trims context, verifies whether the upvote actually happened, detects an unrecovered login redirect, and retries. It then handles login outside the agent's tool loop. The result works without changing the prompt or upgrading the old model. He presents this as a way to do more with less.

### OS-level policies avoid the failure pattern created by permission prompts
[3:38:27](https://www.youtube.com/watch?v=m12vGjfbNlo&t=13107s)
Jun Yu Tan compares coding-agent permission fatigue with SQL injection. Repeated approval prompts interrupt work, so users often disable them, leaving agents with broad access to files, credentials, and environment secrets. Tusk's Fence moves enforcement below the agent. It restricts file paths, filters network access through allowed domains, and checks commands, including nested shells. A policy file defines what the process can see, reach, or run. Tan places this between probabilistic action classification and stronger isolation such as containers or microVMs. The aim is to change what the agent can do rather than repeatedly asking the agent to behave.

### Adaptive systems should optimize data and compute instead of only growing model size
[5:04:48](https://www.youtube.com/watch?v=m12vGjfbNlo&t=18288s)
Sara Hooker argues that the industry has treated larger models as the main route to progress, but the returns from brute-force scaling are weakening. She points to smaller models outperforming larger ones, redundant weights, weight removal, and the value of high-quality data. She describes a future built around adaptive compute, interaction with the world, and continuous learning. Adaption Labs is working on adaptive data and AutoScientists, which automatically learns how to optimize data and models for a task. Her broader claim is that intelligence should adapt across the stack, from data through the interface, rather than forcing everyone to use the same static model.

### Real-world robotics needs useful deployment data and intervention, not only lab performance
[5:20:00](https://www.youtube.com/watch?v=m12vGjfbNlo&t=19200s)
Daniel and Siddharth Krishnan describe teleoperated robots working in insect farms, laundries, food preparation, and hospitality. Teleoperation is expensive and difficult to scale, but it produces data that matches the robot's hardware, environment, and task. Their proposed loop starts with useful commercial work, uses the resulting data for supervised fine-tuning, and then adds tele-supervision when the robot reaches an unreliable plateau. They call 80% autonomy insufficient for production because one failure in five can directly affect a customer. Remote, low-latency supervision can let one operator support multiple robots while corrections become new training data.

### Creative AI needs human taste, iteration, and editable structure
[2:55:36](https://www.youtube.com/watch?v=m12vGjfbNlo&t=10536s)
Josh Newton says AI behaves like a magic pencil and should assist creative judgment rather than replace it. He argues that models tend toward common patterns because they are trained on existing work, so speed can push products toward average quality. His examples include Orbit, a subscription-tracking app shaped by craft and a specific audience, and custom prototyping tools that let him explore states and edge cases quickly. He recommends lateral inspiration, such as starting from a poster rather than another app, and repeated iteration instead of accepting the first prompt. Alex Lee and Priyaa Kalyanaraman extend this argument to design systems and layered graphics, where code, brand rules, editable layers, and specialized models give agents more useful context.

## Notable quotes
- "Planning is the way for your agent to first decide what it is it needs to do before actually actioning on it." (12:24)
- "Connections give access. They don't give understanding." (7:47:09)
- "Stop asking the actor to behave. Change what the actor can do." (3:43:19)
- "AI is a magic pencil. It's time to follow your curiosity and pour yourself into a piece of art." (2:58:49)
- "The hard part is actually not search, it's synthesis." (7:54:08)

## Tools & references mentioned
- Arize AI
- Alex
- Resaro
- Cloudflare
- Code Mode
- MCP
- V8 isolates
- IBM
- OpenClaw
- OpenAI Codex
- Ego AI
- Zo Computer
- PI
- Tavon AI
- Claude Code
- Microsoft AI
- Mastra
- LlamaIndex
- LlamaParse
- LightParse
- Fence
- Adaption Labs
- AutoScientists
- MiniMax
- The Robot Company
- Tesseract.art
- Bifrost
- OpenGraph Labs
- Cortex AI
- Magic Patterns
- MagicPath
- Lica
- Hyperspell
- Lightrint
- Vibe Kanban
- Interfaze AI
- Southbridge
- Smithery
- Ralph loop
- Open RAG
- PassBench
- Hacker News
- GitHub
- Linear
- Together AI
- Terminal Bench
- MMLU
- Gemini
- Gemma
- Veo
- AlphaGenome
- WeatherNext
- Claude
- Cursor
- Figma
- Canva
- Adobe
- Stripe
- Google DeepMind
- OpenAI
- Tusk

## Who should watch
- You are building an agent that loses tasks, overruns its context, or claims success without verifying the result.
- Your team is moving an agent from a demo into production and needs concrete choices around evaluation, permissions, fallbacks, and observability.
- You work on robotics, design tools, company knowledge, or coding infrastructure and want examples of how model behavior can be constrained by better systems.

## Related talks

- [AIE Singapore Day 1](https://aietalks.com/talks/aie-singapore-day-1) (Sherry, 65 Labs & Dr Vivian Balakrishnan, Singapore Ministry of Foreign Affairs & Gavriel Cohen, NanoCo & Thibault Sottiaux, OpenAI & Dr Feng Yuzhang, GovTech Singapore & Phil Hedayatnia, Airfoil & Annie Luo, Google & Keziah & Jimmy Lai, Vercel & Vedran Jukic, Daytona & Vaishant Kameswaran & Rohan Kumar, Greptile & Yuntong Zhang, Sonar & Eugene Cheah, Featherless & Max Buckley, Exa AI & Mark Doyle, Stripe & Li Hau Tan, Simular & Ryo Lu, Cursor & Aosheng Ran, Figma & Selim Arguel, Menlo Research & Alberto Taiuti, Reactor & Jan Liphardt, OpenMind & Andrew Tan, Groq & Daria Soboleva, Cerebras & Zixuan Li, Z.ai & Boris Starkov, ElevenLabs & Jackman Ong, Prime Intellect & Michelle Julia, Bluelabs & Jacky Mok, Reka & Gokul Srinivasan, Antim Labs & Wei Wei Hsu, Lentil & Anun Joshi, Bland & Linh Nguyen, Obello & Stefania Druga, Sakana AI & swyx, Cognition, 10:09:12)
- [Agent Engineering (Day 2)](https://aietalks.com/talks/agent-engineering-day-2) (swyx, AI Engineer Summit & Shashank Kapoors, Princeton University & Mukun Sudar & Arush, Google & Barry Zhang, Anthropic & Zach Renau, Sierra & Will Brown, Morgan Stanley & John Křizí, Jane Street & Anu Koduri, Bloomberg & Mike Conover, Brightwave & Kevin Hou, Windsurf & Mustafa Ali, Method Financial & Kyle Corbett, OpenPipe & Nick Kotakis, SuperDial & Rahul Sanghvi, Ramp & Karina Nguyen, OpenAI & Stefania Druga, Google & Soumith Chintala, Meta and PyTorch, 8:26:36)
- [AI Engineer World's Fair 2025, Day 2 Keynotes & SWE Agents Track](https://aietalks.com/talks/ai-engineer-worlds-fair-2025-day-2-keynotes-swe-agents-track) (Laurie Voss, LlamaIndex & Benjamin Duny, AI Engineer & Logan Kilpatrick & Jack Rae, Google DeepMind & Manu Goyal, Braintrust & Solomon Hykes, Dagger & Jesse Han, Morph & Vibhu Sapra & Scott Wu, Cognition & Rustin Banks, Google Jules & Christopher Harrison, GitHub Copilot & Tomas Reimers, Graphite & Boris Cherny, Anthropic & Robert Brennan, Allhands & Josh Albrecht, Imbue Sculptor & Eno Reyes, Factory & George Cameron, Artificial Analysis & Ankur Goyal, Braintrust & Barr Yaron, Amplify & Alex Atallah, OpenRouter & Sean Grove, OpenAI & Ben & swyx, AI Engineer, 9:08:07)
- [AI Engineer Summit 2023, Day 1 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-1-livestream) (Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai & Amjad Masad & Michele Catasta, Replit & Toran Bruce Richards, AutoGPT & Simón Fishman & Logan Kilpatrick, OpenAI & Flo Crivello, Lindy & Barr Yaron, Amplify & Sasha Sheng & Harrison Chase, LangChain & Shreya Rajpal, Guardrails AI & Eugene Yan, Amazon & Linus Lee, Notion & Brittany Walker, CRV & Chris White, Prefect & Bryan Bischof, Hex, 4:50:00)
- [AI Engineer Summit 2023, Day 2 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-2-livestream) (Mario Rodriguez, GitHub & Dedy Kredo, CodiumAI & Matt Welsh, Fixie.ai & Amelia Wattenberger, Adept & Samantha Whitmore & Jason Yuan, New Computer & Joseph Nelson, Roboflow & Hassan El Mghari, Vercel & Paul Copplestone, Supabase & Daniel Rosenwasser, Microsoft & Jason Liu, Fivesixseven & Anton Troynikov, Chroma & Jerry Liu, LlamaIndex & Mithun Hunsur, Ambient & Abi Aryan, O'Reilly & Simon Willison, Datasette & Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai, 7:30:56)