Alyx escaped context growth by keeping the head and tail of data in the active context while storing the middle in retrievable memory.
2
Long sessions need dedicated evaluations, because agents can appear reliable at first and then forget information much later in a conversation.
3
Sub-agents keep heavy search and analysis data out of the main conversation, while long-term memory and better context-selection metrics remain unfinished work.
Summary
Sally-Ann Delucia describes a year of building Alyx, an agent that analyzes the trace data generated by AI applications. The team found that context management was a product problem as well as an engineering problem. Simple truncation made follow-up questions lose their meaning, while summarization gave the model too much control over what information survived. Their current approach keeps the first and last parts of a context, stores the middle in a memory store, and lets Alyx retrieve it when needed. Long-session evaluations expose failures that only appear after many turns. Sub-agents handle data-heavy searches so the main conversation stays small. Delucia is direct about the remaining gaps: provider limits still matter, long-term memory is missing, and context selection is based on heuristics rather than a principled budget or clear quality metrics.
Delucia says agent teams initially focused on prompt engineering, then learned that context often determines whether an agent succeeds. Context management is more than fitting content under a token limit. The team must choose which information the model should see and which information it can forget. For Alyx, this choice affects the product directly because the agent analyzes traces containing user inputs, prompts, metadata, and later patterns across many traces. Delucia describes context management as a product and UX problem as well as an engineering problem. If the agent receives the wrong data, it gives poor answers and users stop relying on it.
Alyx was trapped by the trace data it was built to analyze
The team built Alyx using Alyx, hoping that an agent that helped them build their application would also be useful to customers. That created a feedback loop. Alyx ran on trace and span data, the spans grew, the context limit was reached, and Alyx failed. Because the failed run was itself recorded in the span data, the retry added more data and failed again. Delucia describes the system as being constrained by the data it needed to understand. Escaping the loop required separating active context from memory and moving heavy work away from one agent.
Simple truncation caused the agent to forget the conversation
The first attempt took only the beginning of a long context, initially the first 100 characters, and dropped everything else. It worked for simple requests until follow-up questions exposed the problem. After Alyx answered a question about common inputs, it could not understand a follow-up about input B because the earlier information had disappeared. Delucia says over-truncation broke reasoning. The problem was not only missing detail in one answer. Follow-ups began to look like new conversations, so the agent lost the references needed to continue its analysis.
Summarization gave the model too much control over what survived
The next idea was to ask an LLM to compress the full context into fewer tokens. Delucia calls this the obvious solution, but the team found it inconsistent. The summarizer decided what mattered, and Alyx had no reliable way to control that decision. Information needed for a later query could disappear even if it seemed unimportant during summarization. The team therefore abandoned summarization as its main context strategy. Delucia says she was surprised that this approach did not work, since current LLMs are good at producing summaries, but quality summaries did not provide the control Alyx needed.
Head-tail preservation with retrieval became Alyx's working strategy
Alyx now keeps the first 100 characters and the last 100 characters of a context, removes the middle from the active prompt, and stores that middle in memory. The system keeps the system prompt, retains the latest result from long tool calls, and avoids repeated messages where possible. When Alyx decides that an older tool call or message matters, it can retrieve that context from the memory store. This gives the active context a small fixed shape while preserving access to the full history. Delucia says the combination has worked for months, although the team is revisiting it as sessions become longer.
Long-session evaluations expose failures before users do
Alyx users often stay in one chat while moving through different pages of the application. That makes conversations grow, and failures can appear late. The team initially saw the smart-truncation approach working, then found that Alyx began forgetting information far into longer sessions. Their response was to create long-session evaluations. They load 10 turns and test the 11th, which makes late context failures reproducible instead of something found through a user report or manual inspection. Delucia presents these evaluations as a useful signal for judging context management, since long sessions occur naturally in agent applications.
Sub-agents keep heavy analysis out of the main conversation
Alyx's search tasks can involve hundreds of spans, multiple queries, large data sets, and substantial intermediate reasoning. The team decided that all of this did not belong in the main chat context. The main agent now keeps the conversation and light context, then delegates data-heavy work to a sub-agent. The sub-agent holds the search data and intermediate context, returns a result to the main agent, and can use the memory store when it needs older information. Delucia says the team has rolled out many sub-agents after learning that heavy operations could be separated from the user's continuing conversation.
Long-term memory and principled context budgets are still missing
Delucia is clear that the current design does not solve every context problem. Very large prompts and inputs can still hit provider limits, especially because Alyx's customers are asking it to understand the agent data that includes those prompts and conversations. The team keeps returning to sub-agents as a way to split the load. Alyx also lacks long-term memory across chats. Its current memory store helps retrieve compressed conversation context, but it does not let users refer reliably to issues discussed in an earlier chat. Context selection remains a heuristic, such as keeping the first and last 100 characters, without a clear context budget or direct quality metrics.
