# From Context to Memory: Your Agents Need a Real Memory Layer Anders Swanson, Oracle | AI Engineer | 18:20 Source: https://www.youtube.com/watch?v=BIhiYL4U9_M Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/from-context-to-memory-your-agents-need-a-real-memory-layer Published: 2026-10-09 Tags: memory, rag, search ## TL;DR - Context windows, markdown files, and scratch pads do not give agents persistent, governed memory. - A useful memory layer forms typed memories, stores them with vectors, text indexes, graph links, time data, and audit history, then recalls them with hybrid search. - Memory systems need redaction, scoping, consolidation, review, and feedback so stale, poisoned, conflicting, or irrelevant memories do not control later work. ## Summary Anders Swanson treats agent memory as a storage and retrieval system rather than a larger prompt. An agent session can produce a useful procedure or fact, but that work disappears unless it is deliberately formed into managed state. He describes four memory types: episodic, semantic, procedural, and working state. A memory record can include text, JSON metadata, embeddings, full-text indexes, graph links, temporal fields, and audit events. Formation starts with redaction before enrichment or embedding, followed by scope checks and provenance. Recall uses hybrid search across vectors, lexical matches, entities, graphs, time, and feedback signals. The system must also consolidate duplicates, detect conflicts, retire stale facts, and measure whether retrieved memories helped or caused harm. The talk is practical and closely tied to Oracle's multi-model database and agent-memory SDK. ## Key ideas ### A context window records the current run, but it does not preserve experience [00:55](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=55s) An agent may read documents, run queries, reproduce a failure, and discover a useful procedure during a support session. When the session ends, that work is discarded if no memory layer saves it. A later agent then repeats the searches and queries, creating more chances for failure and hallucination. Swanson says memory begins only when useful experience becomes managed state. A larger context window does not solve this because long sessions accumulate irrelevant information and compaction can remove useful signal. ### Files and scratch pads become an unmanaged database [01:56](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=116s) Markdown files, wikis, prompts, and file hierarchies can help agents carry information between sessions, but Swanson does not consider them governed recall. Files do not scale well in large multi-tenant environments, and grep cannot provide complex hybrid retrieval. His practical warning is that continuing to build a memory layer on top of files eventually means building a database anyway. The alternative is to use a storage system designed for structured records, search, relationships, and access controls. ### A memory system has formation, recall, evolution, evaluation, and governance [03:04](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=184s) Swanson describes a pattern that combines deliberate memory formation with retrieval, consolidation, and measurement. Formation creates distilled experiences and structured facts rather than storing raw transcript dumps. Recall combines several search methods. Evolution handles duplicates, conflicts, and changing validity over time. Evaluation supplies feedback about use and usefulness. Governance redacts secrets and personal information, scopes rights to agents, workflows, teams, or users, and provides review and suppression controls for stale or poisoned data. ### Typed memories let the system store and search different kinds of knowledge [06:22](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=382s) The model separates episodic memory, semantic memory, procedural memory, and working state. Episodic memory contains specific past events or transcript slices. Semantic memory contains facts about workflows, users, and other entities. Procedural memory records reusable ways to perform tasks, such as a build step. Working state captures short-term information about the current task and may expire or later be promoted into another memory type. These categories also allow retrieval to use different search methods for different records. ### Memory records need several lookup paths and an audit trail [07:17](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=437s) A memory object can have an ID, text content, JSON tags, document data, and a vector embedding for similarity search. Swanson also recommends a text index for keyword search, graph links for related entities and memories, temporal fields for validity and last use, and audit events that record who created the memory and how it was used. The record is compact, while an episode can preserve the transcript slice that produced it. That relationship provides provenance and extra context during later recall. ### Safe formation starts with redaction and scope checks [08:26](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=506s) Before enrichment or embedding, the system should remove personally identifying information, secrets, and irrelevant material. It can then add metadata, apply agent, team, and organization scope, create an embedding, and link the memory to the episode that produced it. Organization-specific policies can be applied before the record reaches storage. Swanson treats redaction as a first step because sensitive data becomes harder to control after it has entered embeddings or other derived records. ### Hybrid retrieval combines evidence instead of trusting one similarity score [10:02](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=602s) Recall is a query against the storage layer. Hybrid search can combine vector similarity, full-text search, relational entity matches, graph traversal, temporal signals, and feedback such as importance or usefulness. The system can fuse results, apply thresholds, re-rank them, and return a context card with scored evidence and provenance. Exact fact lookups may need an ID match rather than semantic similarity. The weights for each search branch can be tuned, and the final result can be limited to the highest-ranked memories. ### Feedback determines whether memories should remain, change, or be removed [12:17](https://www.youtube.com/watch?v=BIhiYL4U9_M&t=737s) Memory cannot be a passive repository. It must handle stale facts, bad scope, poisoned instructions, conflicting records, over-recall, irrelevant matches, data leakage, and silent drift as the collection grows. Swanson gives a build-system change as an example of stale memory: advice about NPM can become wrong after a project switches systems. Retrieval outcomes should be recorded as helpful, stale, irrelevant, or harmful. That feedback can change ranking and lifecycle decisions, including promoting useful memories or lowering harmful ones. ## Notable quotes - "Context windows are not memory." (00:55) - "And I feel like if we continue to the logical conclusion of trying to build a memory layer on top of files, we're going to be building our own database." (02:43) - "We should first start with redaction." (08:26) - "Feedback is what closes the loop from memory." (15:25) ## Tools & references mentioned - Claude Code - Codex - Oracle Database - Oracle Agent Memory Python SDK - Oracle AI Developer Hub - MCP ## Who should watch - You are building agents that repeat file reads, queries, or procedures across sessions and need a persistent state model. - Your team is deciding whether markdown files, prompts, or a vector index are enough for memory in a multi-tenant system. - You need to design retrieval and governance together, especially when memories may contain personal data, secrets, conflicting instructions, or stale project practices. ## Related talks - [Architecting Agent Memory: Principles, Patterns, and Best Practices](https://aietalks.com/talks/architecting-agent-memory-principles-patterns-and-best-practices) (Richmond Alake, MongoDB, 17:37) - [Giving AI Agents Memory That Learns](https://aietalks.com/talks/giving-ai-agents-memory-that-learns) (Jake Broekhuizen, LangChain, 16:39) - [Memory Masterclass: Make Your AI Agents Remember What They Do!](https://aietalks.com/talks/memory-masterclass-make-your-ai-agents-remember-what-they-do) (Mark Bain, AIUS & Vasilia Marovitz, Cognee & Daniel Chalef, Graphiti and Zep & Alex Gilmore, Neo4j, 51:25) - [How We Solved Context Management in Agents](https://aietalks.com/talks/how-we-solved-context-management-in-agents) (Sally-Ann DeLucia, Arize AI, 16:17) - [Stateful Agents](https://aietalks.com/talks/stateful-agents) (Charles Packer, Letta, 1:19:34)