# How We Solved Context Management in Agents Sally-Ann Delucia, Arize | AI Engineer Europe 2026 | 16:17 Source: https://www.youtube.com/watch?v=esY99nYXxR4 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/how-we-solved-context-management-in-agents Published: 2026-05-10 Tags: agents, context-engineering, evals, memory, multi-agent ## TL;DR - Alyx escaped context growth by keeping the head and tail of data in the active context while storing the middle in retrievable memory. - Long sessions need dedicated evaluations, because agents can appear reliable at first and then forget information much later in a conversation. - Sub-agents keep heavy search and analysis data out of the main conversation, while long-term memory and better context-selection metrics remain unfinished work. ## Summary Sally-Ann Delucia describes a year of building Alyx, an agent that analyzes the trace data generated by AI applications. The team found that context management was a product problem as well as an engineering problem. Simple truncation made follow-up questions lose their meaning, while summarization gave the model too much control over what information survived. Their current approach keeps the first and last parts of a context, stores the middle in a memory store, and lets Alyx retrieve it when needed. Long-session evaluations expose failures that only appear after many turns. Sub-agents handle data-heavy searches so the main conversation stays small. Delucia is direct about the remaining gaps: provider limits still matter, long-term memory is missing, and context selection is based on heuristics rather than a principled budget or clear quality metrics. ## Key ideas ### Context engineering decides what the model sees [01:52](https://www.youtube.com/watch?v=esY99nYXxR4&t=112s) Delucia says agent teams initially focused on prompt engineering, then learned that context often determines whether an agent succeeds. Context management is more than fitting content under a token limit. The team must choose which information the model should see and which information it can forget. For Alyx, this choice affects the product directly because the agent analyzes traces containing user inputs, prompts, metadata, and later patterns across many traces. Delucia describes context management as a product and UX problem as well as an engineering problem. If the agent receives the wrong data, it gives poor answers and users stop relying on it. ### Alyx was trapped by the trace data it was built to analyze [03:57](https://www.youtube.com/watch?v=esY99nYXxR4&t=237s) The team built Alyx using Alyx, hoping that an agent that helped them build their application would also be useful to customers. That created a feedback loop. Alyx ran on trace and span data, the spans grew, the context limit was reached, and Alyx failed. Because the failed run was itself recorded in the span data, the retry added more data and failed again. Delucia describes the system as being constrained by the data it needed to understand. Escaping the loop required separating active context from memory and moving heavy work away from one agent. ### Simple truncation caused the agent to forget the conversation [05:13](https://www.youtube.com/watch?v=esY99nYXxR4&t=313s) The first attempt took only the beginning of a long context, initially the first 100 characters, and dropped everything else. It worked for simple requests until follow-up questions exposed the problem. After Alyx answered a question about common inputs, it could not understand a follow-up about input B because the earlier information had disappeared. Delucia says over-truncation broke reasoning. The problem was not only missing detail in one answer. Follow-ups began to look like new conversations, so the agent lost the references needed to continue its analysis. ### Summarization gave the model too much control over what survived [06:18](https://www.youtube.com/watch?v=esY99nYXxR4&t=378s) The next idea was to ask an LLM to compress the full context into fewer tokens. Delucia calls this the obvious solution, but the team found it inconsistent. The summarizer decided what mattered, and Alyx had no reliable way to control that decision. Information needed for a later query could disappear even if it seemed unimportant during summarization. The team therefore abandoned summarization as its main context strategy. Delucia says she was surprised that this approach did not work, since current LLMs are good at producing summaries, but quality summaries did not provide the control Alyx needed. ### Head-tail preservation with retrieval became Alyx's working strategy [06:46](https://www.youtube.com/watch?v=esY99nYXxR4&t=406s) Alyx now keeps the first 100 characters and the last 100 characters of a context, removes the middle from the active prompt, and stores that middle in memory. The system keeps the system prompt, retains the latest result from long tool calls, and avoids repeated messages where possible. When Alyx decides that an older tool call or message matters, it can retrieve that context from the memory store. This gives the active context a small fixed shape while preserving access to the full history. Delucia says the combination has worked for months, although the team is revisiting it as sessions become longer. ### Long-session evaluations expose failures before users do [08:02](https://www.youtube.com/watch?v=esY99nYXxR4&t=482s) Alyx users often stay in one chat while moving through different pages of the application. That makes conversations grow, and failures can appear late. The team initially saw the smart-truncation approach working, then found that Alyx began forgetting information far into longer sessions. Their response was to create long-session evaluations. They load 10 turns and test the 11th, which makes late context failures reproducible instead of something found through a user report or manual inspection. Delucia presents these evaluations as a useful signal for judging context management, since long sessions occur naturally in agent applications. ### Sub-agents keep heavy analysis out of the main conversation [09:23](https://www.youtube.com/watch?v=esY99nYXxR4&t=563s) Alyx's search tasks can involve hundreds of spans, multiple queries, large data sets, and substantial intermediate reasoning. The team decided that all of this did not belong in the main chat context. The main agent now keeps the conversation and light context, then delegates data-heavy work to a sub-agent. The sub-agent holds the search data and intermediate context, returns a result to the main agent, and can use the memory store when it needs older information. Delucia says the team has rolled out many sub-agents after learning that heavy operations could be separated from the user's continuing conversation. ### Long-term memory and principled context budgets are still missing [11:19](https://www.youtube.com/watch?v=esY99nYXxR4&t=679s) Delucia is clear that the current design does not solve every context problem. Very large prompts and inputs can still hit provider limits, especially because Alyx's customers are asking it to understand the agent data that includes those prompts and conversations. The team keeps returning to sub-agents as a way to split the load. Alyx also lacks long-term memory across chats. Its current memory store helps retrieve compressed conversation context, but it does not let users refer reliably to issues discussed in an earlier chat. Context selection remains a heuristic, such as keeping the first and last 100 characters, without a clear context budget or direct quality metrics. ## Notable quotes - "The best context strategy is one that lets your agents remember what it needs to and forget what it doesn't." (02:09) - "Over truncation had broke the reasoning. It couldn't remember." (06:00) - "The main conversation can stay small." (10:15) - "Agents don't fail because of prompts, they fail because of context." (14:08) ## Tools & references mentioned - Alyx - Arize - Andre Karpathy - Claude Code - Claude - Cursor ## Who should watch - You are building an agent that reads traces, documents, tool results, or other data that grows faster than the conversation window. - Your agent seems reliable in short tests but starts forgetting references after users stay in one chat for a long time. - You are deciding between truncation, summarization, retrieval, and sub-agents and want implementation lessons from a system in production development. ## Editor's note From the pack [Context engineering](https://aietalks.com/packs/context-engineering): Delucia tests failures late in long sessions, Hablich measures tokens per successful outcome, and Pai insists that generated code remain inspectable. Those checks need records of real runs. Kitaru captures an agent's inputs, outputs and tool calls so a team can replay the same task after changing retrieval, compaction or tool exposure and inspect what the new context changed. Written by the AIE Talks editors (the Kitaru team), not by the speaker. ## Related talks - [Why More Context Makes Your Agent Dumber and What to Do About It](https://aietalks.com/talks/why-more-context-makes-your-agent-dumber-and-what-to-do-about-it) (Nupur Sharma, Qodo, 26:27) - [Context Engineering in 2026](https://aietalks.com/talks/context-engineering-in-2026) (Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI, 1:03:26) - [Stateful Agents](https://aietalks.com/talks/stateful-agents) (Charles Packer, Letta, 1:19:34) - [Build Agents That Run for Hours](https://aietalks.com/talks/build-agents-that-run-for-hours) (Ash Prabaker & Andrew Wilson, Anthropic, 1:15:40) - [Claude for Long-Horizon Tasks](https://aietalks.com/talks/claude-for-long-horizon-tasks) (Lance Martin, Anthropic, 25:19)