Context compaction can cost more and reduce recall when prompt caching makes resending the full history cheap.
2
In the AI tutor experiments, leaving the conversation history untouched beat compaction on recall, cost, and latency.
3
The right context strategy depends on the constraint, since local hardware, long documents, and cloud caching lead to different choices.
Summary
Louis-François Bouchard, Omar Solano, and Samridhi Vaid describe the context problems behind Towards AI's open-source AI tutor. The model has a finite context window and no memory between sessions, while tool outputs, retrieved lessons, code, and chat history accumulate over time. They compare trimming, summarization, selective retention, retrieval, file browsing, and full-history approaches. Prompt caching changes the cost calculation: transforming a history can invalidate cached tokens, while resending the same history can be cheap and fast. In their tests, full history produced better recall and lower cost and latency than their production defaults. The team also found that a browsing tool returned the same recall as hybrid retrieval while taking longer. Local models required retrieval or compaction because the tested hardware supported only a 32k context window. Their final tutor setup uses DeepSeek V4 Flash, hybrid retrieval, and full history until a 30k-token limit.
Context growth can hurt answer quality, cost, and latency
Louis-François Bouchard says agent failures often come from an overloaded context rather than from a model that suddenly became less capable. The tutor's context contains its instructions, course material, tool definitions, chat history, old tool outputs, retrieved course chunks, and the current question. Old tool calls and results are the main source of growth. A larger context can degrade quality because language models must handle many injected facts at once. It also raises spending and time to first token because earlier tokens are sent again on every turn. The tutor therefore needs context management within a session, separate from memory across sessions.
Compaction can start with simple rules before using another model
The team defines compaction as keeping the smallest context that still contains what the model needs, while dropping or saving the rest. Some methods need no language model. A system can truncate unusually large tool outputs, retain only the beginning and end, use a sliding window over recent turns, or clear outputs from tools whose results are no longer useful. Smaller local models can also summarize or decide which parts to retain. The techniques they discuss include selective retention, continuous summarization, reset-after-summary approaches, and delta summarization. Delta summarization is useful when a subagent passes an evolving summary to a main agent, although the tutor uses one main agent instead.
Instead of deleting information, the tutor can save it in files, memory, or a retrieval system. The team's LLM wiki design stores content in chunks, cross-links those chunks, and points back to raw files. An index gives the agent a small map of what exists. The agent can read an index entry, inspect a chunk, and open the raw source only when the task needs more detail. This lets simple questions pull little context and complex questions pull more. Bouchard also recommends smaller, precise skills that refer to one another, so the system loads only the skill context needed for the current task.
Prompt caching makes summarization a potentially expensive transformation
Prompt caching lets providers reuse computation for tokens that were already sent. Bouchard says cached tokens can cost far less than new tokens, with DeepSeek offering a discount of up to 50 times in the experiments. A summary or other rewritten context changes the token sequence, so the provider cannot reuse the old cache and charges the transformed tokens at the full rate. He estimates that compaction must shrink the context by more than 50 times before it pays for itself under that discount, without counting possible quality loss. His guidance is to clear stale sessions or tool outputs, use progressive disclosure, compact selectively, and optimize for cache hits rather than summarizing every conversation.
The tutor uses one ReAct agent with hybrid retrieval
Omar Solano presents a simple architecture built around one ReAct-style agent that loops over thinking blocks and tool calls. LangChain creates the agent with an in-memory saver, and middleware can clear tool outputs or summarize the history at runtime. The tutor's retrieval tool searches an 8-million-token corpus containing Towards AI lessons and documentation from projects such as LangChain, LlamaIndex, OpenAI, and Claude Code. It combines semantic search with BM25, merges the results, reranks them, and returns the top five chunks. The system also limits retrieved context to 100,000 tokens and lets students select the sources used for an answer.
Knowledge-base browsing added latency without improving recall
The team added a second tool that lets the agent browse a read-only knowledge base with shell commands. The files include raw markdown, generated titles, and a concise wiki that groups related material. The tool is sandboxed to the knowledge-base directory, limits commands to 20 per turn, returns an error after eight seconds, and caps tool output at 40,000 characters. Although the agent used the browsing tool in almost 90% of turns, experiments found no recall improvement over the original hybrid retrieval tool. Recall stayed the same while the browsing version ran 50% slower because it made more tool calls. The team kept the experiment available but did not find a reason to use it in the deployed setup.
Omar describes 11 presets evaluated with the same model, prompt, tools, and data. The tests used real student questions for single-turn tasks and multi-turn conversations with facts inserted early in the history. The original production settings cleared tool outputs after 5,000 tokens and summarized after 30,000 tokens. In the multi-turn tests, leaving the history untouched produced better recall than those defaults. It was also cheaper and faster. Removing tool outputs often forced the agent to retrieve information it had already seen, which added tool calls, tokens, and latency. The results came from a small set of 11-to-13-turn conversations, so the team ran follow-up tests with cheaper models and longer contexts.
Cloud caching preserved details that summaries lost
Samridhi Vaid tests whether full history preserves specific facts in realistic student conversations. With DeepSeek V4 Flash, the model returned the exact requested details 95% of the time when the full context remained intact. After summarization or compaction, it returned the details only 32% of the time. In another long-context test, distinctive facts remained available up to 800,000 tokens, while more ambiguous facts performed worse. The full-history setup also sent the most tokens but was the cheapest because 97% of those tokens were cached. A 36-turn conversation reached about 1.78 million tokens, yet keeping the history still came out ahead in the tested configuration.
The answer changes on local hardware. On a MacBook, the tested models supported a maximum 32k-token context window, so some individual lessons were already too large and full history could not continue indefinitely. Moving from a 7B or 8B model to a 32B model did not increase the available context window. For oversized pasted documents, retrieval worked well locally, with reported accuracy of 100% and processing times between 25 and 65 seconds. Stuffing the window with excessive history took about 340 seconds and produced only a single-token output in one test. Dense retrieval also failed when facts were buried at 400k tokens, reaching 0% recall, while BM25 continued to find them.
"You have to name the constraint that you have and then look for a better alternative."1:01:40
Who should watch
You are building a long-running agent and currently summarize or trim every conversation without measuring recall, cache hits, cost, and latency.
Your application mixes chat history with tool results, retrieved documents, code, or logs, and you need to decide which information belongs in the active context.
You are choosing between a cloud model and a local model, since the talk shows how context-window and hardware limits change the best design.
