# We Kept Rebuilding Our AI Agent. Here's What Survived

Fuad Ali, Arize AI | AI Engineer NYC 2026 | 20:06

Source: https://www.youtube.com/watch?v=YIhvrvHtonk
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/we-kept-rebuilding-our-ai-agent-heres-what-survived
Published: 2026-10-27
Tags: evals, harness-engineering, memory, observability, tracing

## TL;DR
- Alex improved mainly through changes to its harness code, including its planning loop, stopping rule, memory handling, and skill loading.
- Large tool results are stored like files, while each user's long-term memory is kept in one auditable Markdown file capped at 8,000 characters.
- Production sessions become regression cases, but people still decide what quality means, how much speed to trade for quality, and which fixes are acceptable.

## Summary
Fuad Ali describes how Arize's in-product agent Alex changed from a late-2023 prototype that hallucinated about half the time into a single agent with planning, memory, and on-demand skills. Arize wrote its own harness instead of using an agent framework. The harness owns the plan, keeps state in messages, enforces an explicit finish call, and limits how much history enters the context. Large tool results are truncated and referenced as files. Long-term memory is one Markdown file per user, capped at 8,000 characters, written by a background job after a session. Arize also merged ten specialist agents into one agent that loads skills when needed. Live production sessions become regression cases and can be replayed in experiments. Ali is clear about what remains human work: creating gold labels, setting quality baselines, and deciding when speed or model changes are worth a quality trade-off.

## Key ideas
### Alex moved from a router and tool loop to a planned production agent
[01:59](https://www.youtube.com/watch?v=YIhvrvHtonk&t=119s)
The first Alex prototype in late 2023 was a router over skills and tool calls. A staff engineer described it as about 50% hallucination. The system later gained planning, a richer interface, long-term memory, and the ability to explain its work. Alex can read traces, turn findings into tests, build datasets and experiments, and create synthetic data. Ali says the team rebuilt the surrounding harness rather than training the models themselves.

### The harness gradually took ownership of the plan
[06:19](https://www.youtube.com/watch?v=YIhvrvHtonk&t=379s)
When the model owned the to-do list, it spent many calls checking items off instead of doing useful work. It also lost focus after repeated tool calls. Arize first added an in-progress marker, then rendered the plan in every model message so the rules appeared beside the current state. The marker was later removed because unfinished work already had a ranked order. The team was testing a design with no to-do tool, with the harness maintaining state and the model doing the next task.

### Finishing became an explicit gate in the harness
[08:33](https://www.youtube.com/watch?v=YIhvrvHtonk&t=513s)
Arize initially treated a response without tool calls as completion. That failed when some models announced a step in one turn and made the tool call in the next. Alex now has an explicit finish call. If plan items remain open, the harness can send the work back up to three times. It also warns one step before the iteration limit and can force a finish. This protects the agent across different customer-selected models.

### Large tool results are handled as files instead of context
[09:44](https://www.youtube.com/watch?v=YIhvrvHtonk&t=584s)
Ali compares oversized tool results to books covering a desk. A large JSON trace can consume the context window and leave no room for useful work. Alex truncates the JSON, keeps a pointer and a high-level description, and lets the agent retrieve the full result with jq when needed. The harness also writes checkpoints when the agent's trajectory grows too long and drops replaced history within a fixed token budget.

### One small memory file beat more elaborate storage systems
[12:10](https://www.youtube.com/watch?v=YIhvrvHtonk&t=730s)
Arize tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. It settled on one Markdown file per user, capped at 8,000 characters. The file loads on every full request and is updated by a background job after the session. Ali values that it is small, easy to load, and easy for a human to audit. A human labeler can inspect the file and help adjust the background job's instructions.

### One agent with on-demand skills replaced ten specialists
[13:33](https://www.youtube.com/watch?v=YIhvrvHtonk&t=813s)
Alex first had a trace-analysis agent, then ten specialists tied to different product pages. Arize later merged them into one agent that loads skill instructions and tools when needed. Ali connects this change to improved model capability and to a simpler design. The agent decides which tools to use instead of being routed into a page-specific specialist.

### A fast, good-enough model can beat a newer default
[14:13](https://www.youtube.com/watch?v=YIhvrvHtonk&t=853s)
Ali rejects the idea that the newest model is automatically the best default. The best general frontier model may not be the best choice for a particular product surface. Arize usually chooses a model that is fast and good enough, then spends more effort on the harness. The team is exploring routing models by task and small models trained for narrow surfaces. Some tasks, such as generating filters, may need only one model call and no agent loop.

### Production sessions become living regression fixtures
[15:28](https://www.youtube.com/watch?v=YIhvrvHtonk&t=928s)
Arize traces every Alex interaction. User sessions can become regression datasets, and a complete session can be stored so a particular turn can be replayed. These fixtures grow as the team finds new failure cases. Experiments compare harness changes. Ali says agent bugs often appear as qualitative problems rather than explicit errors, such as 43 identical tool calls in one turn or a response that takes 227 seconds.

### People still set the quality baseline and make trade-offs
[17:46](https://www.youtube.com/watch?v=YIhvrvHtonk&t=1066s)
Human judgment remains in the gold labels, golden datasets, and evaluator baselines. People decide how much speed can be traded for quality and whether a model change is worth higher latency. Metrics can show the expected movement in latency and response quality, but they do not make that product decision automatically. Ali also says judges need a human baseline so their evaluations match what engineers consider acceptable.

## Notable quotes
- "We don't use an agent framework. The loop, the plan, the stopping rule, and the memory are all ours." (06:02)
- "Treat a huge result like a file and just scrap it. Don't dump it all into context. Don't worry about doing anything fancy." (12:11)
- "The newest model isn't always the best by default." (14:09)
- "Those production sessions become our test cases and become our living fixtures over time." (16:05)
- "Bugs rarely show up as explicit errors." (16:28)

## Tools & references mentioned
- Alex
- Arize AI
- Arize AX
- Signal
- Anthropic
- OpenAI
- Sonnet 4.5
- jq
- vector retrieval
- episodic stores
- semantic stores
- temporal knowledge graphs

## Who should watch
- You are building an agent harness and need concrete decisions about planning, stopping, tool-result size, or memory.
- Your agent has long traces and vague failures, and you want to turn real user sessions into regression cases.
- You are choosing between specialist agents and one general agent, or deciding whether a newer model is worth its cost and latency.

## Editor's note

Fuad Ali says Arize traces every Alex interaction and turns user sessions into regression datasets when the team finds new failure cases. Kitaru records each agent run with its model calls and tool results, then replays the run with the same inputs and responses, so a production failure can be tested again after a change.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [How We Built an Agent That Improves Itself](https://aietalks.com/talks/how-we-built-an-agent-that-improves-itself) (Zubin Aysola, Weights & Biases, 17:06)
- [Fighting AI with AI](https://aietalks.com/talks/fighting-ai-with-ai) (Lawrence Jones, Incident.io, 17:29)
- [We Trust AI Agents That Break Every Day](https://aietalks.com/talks/we-trust-ai-agents-that-break-every-day) (Shy Ruparel, Temporal, 21:32)
- [Your Agent Fix Just Broke Something Else](https://aietalks.com/talks/your-agent-fix-just-broke-something-else) (Sofia Jakovcevic, Arize AI, 19:25)
- [Agents That Self Improve While You Sleep](https://aietalks.com/talks/agents-that-self-improve-while-you-sleep) (Fuad Ali, Arize AI, 53:05)
