Alex improved mainly through changes to its harness code, including its planning loop, stopping rule, memory handling, and skill loading.
2
Large tool results are stored like files, while each user's long-term memory is kept in one auditable Markdown file capped at 8,000 characters.
3
Production sessions become regression cases, but people still decide what quality means, how much speed to trade for quality, and which fixes are acceptable.
Summary
Fuad Ali describes how Arize's in-product agent Alex changed from a late-2023 prototype that hallucinated about half the time into a single agent with planning, memory, and on-demand skills. Arize wrote its own harness instead of using an agent framework. The harness owns the plan, keeps state in messages, enforces an explicit finish call, and limits how much history enters the context. Large tool results are truncated and referenced as files. Long-term memory is one Markdown file per user, capped at 8,000 characters, written by a background job after a session. Arize also merged ten specialist agents into one agent that loads skills when needed. Live production sessions become regression cases and can be replayed in experiments. Ali is clear about what remains human work: creating gold labels, setting quality baselines, and deciding when speed or model changes are worth a quality trade-off.
Alex moved from a router and tool loop to a planned production agent
The first Alex prototype in late 2023 was a router over skills and tool calls. A staff engineer described it as about 50% hallucination. The system later gained planning, a richer interface, long-term memory, and the ability to explain its work. Alex can read traces, turn findings into tests, build datasets and experiments, and create synthetic data. Ali says the team rebuilt the surrounding harness rather than training the models themselves.
When the model owned the to-do list, it spent many calls checking items off instead of doing useful work. It also lost focus after repeated tool calls. Arize first added an in-progress marker, then rendered the plan in every model message so the rules appeared beside the current state. The marker was later removed because unfinished work already had a ranked order. The team was testing a design with no to-do tool, with the harness maintaining state and the model doing the next task.
Arize initially treated a response without tool calls as completion. That failed when some models announced a step in one turn and made the tool call in the next. Alex now has an explicit finish call. If plan items remain open, the harness can send the work back up to three times. It also warns one step before the iteration limit and can force a finish. This protects the agent across different customer-selected models.
Large tool results are handled as files instead of context
Ali compares oversized tool results to books covering a desk. A large JSON trace can consume the context window and leave no room for useful work. Alex truncates the JSON, keeps a pointer and a high-level description, and lets the agent retrieve the full result with jq when needed. The harness also writes checkpoints when the agent's trajectory grows too long and drops replaced history within a fixed token budget.
One small memory file beat more elaborate storage systems
Arize tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. It settled on one Markdown file per user, capped at 8,000 characters. The file loads on every full request and is updated by a background job after the session. Ali values that it is small, easy to load, and easy for a human to audit. A human labeler can inspect the file and help adjust the background job's instructions.
One agent with on-demand skills replaced ten specialists
Alex first had a trace-analysis agent, then ten specialists tied to different product pages. Arize later merged them into one agent that loads skill instructions and tools when needed. Ali connects this change to improved model capability and to a simpler design. The agent decides which tools to use instead of being routed into a page-specific specialist.
A fast, good-enough model can beat a newer default
Ali rejects the idea that the newest model is automatically the best default. The best general frontier model may not be the best choice for a particular product surface. Arize usually chooses a model that is fast and good enough, then spends more effort on the harness. The team is exploring routing models by task and small models trained for narrow surfaces. Some tasks, such as generating filters, may need only one model call and no agent loop.
Production sessions become living regression fixtures
Arize traces every Alex interaction. User sessions can become regression datasets, and a complete session can be stored so a particular turn can be replayed. These fixtures grow as the team finds new failure cases. Experiments compare harness changes. Ali says agent bugs often appear as qualitative problems rather than explicit errors, such as 43 identical tool calls in one turn or a response that takes 227 seconds.
People still set the quality baseline and make trade-offs
Human judgment remains in the gold labels, golden datasets, and evaluator baselines. People decide how much speed can be traded for quality and whether a model change is worth higher latency. Metrics can show the expected movement in latency and response quality, but they do not make that product decision automatically. Ali also says judges need a human baseline so their evaluations match what engineers consider acceptable.