Context windows, markdown files, and scratch pads do not give agents persistent, governed memory.
2
A useful memory layer forms typed memories, stores them with vectors, text indexes, graph links, time data, and audit history, then recalls them with hybrid search.
3
Memory systems need redaction, scoping, consolidation, review, and feedback so stale, poisoned, conflicting, or irrelevant memories do not control later work.
Summary
Anders Swanson treats agent memory as a storage and retrieval system rather than a larger prompt. An agent session can produce a useful procedure or fact, but that work disappears unless it is deliberately formed into managed state. He describes four memory types: episodic, semantic, procedural, and working state. A memory record can include text, JSON metadata, embeddings, full-text indexes, graph links, temporal fields, and audit events. Formation starts with redaction before enrichment or embedding, followed by scope checks and provenance. Recall uses hybrid search across vectors, lexical matches, entities, graphs, time, and feedback signals. The system must also consolidate duplicates, detect conflicts, retire stale facts, and measure whether retrieved memories helped or caused harm. The talk is practical and closely tied to Oracle's multi-model database and agent-memory SDK.
A context window records the current run, but it does not preserve experience
An agent may read documents, run queries, reproduce a failure, and discover a useful procedure during a support session. When the session ends, that work is discarded if no memory layer saves it. A later agent then repeats the searches and queries, creating more chances for failure and hallucination. Swanson says memory begins only when useful experience becomes managed state. A larger context window does not solve this because long sessions accumulate irrelevant information and compaction can remove useful signal.
Files and scratch pads become an unmanaged database
Markdown files, wikis, prompts, and file hierarchies can help agents carry information between sessions, but Swanson does not consider them governed recall. Files do not scale well in large multi-tenant environments, and grep cannot provide complex hybrid retrieval. His practical warning is that continuing to build a memory layer on top of files eventually means building a database anyway. The alternative is to use a storage system designed for structured records, search, relationships, and access controls.
A memory system has formation, recall, evolution, evaluation, and governance
Swanson describes a pattern that combines deliberate memory formation with retrieval, consolidation, and measurement. Formation creates distilled experiences and structured facts rather than storing raw transcript dumps. Recall combines several search methods. Evolution handles duplicates, conflicts, and changing validity over time. Evaluation supplies feedback about use and usefulness. Governance redacts secrets and personal information, scopes rights to agents, workflows, teams, or users, and provides review and suppression controls for stale or poisoned data.
Typed memories let the system store and search different kinds of knowledge
The model separates episodic memory, semantic memory, procedural memory, and working state. Episodic memory contains specific past events or transcript slices. Semantic memory contains facts about workflows, users, and other entities. Procedural memory records reusable ways to perform tasks, such as a build step. Working state captures short-term information about the current task and may expire or later be promoted into another memory type. These categories also allow retrieval to use different search methods for different records.
Memory records need several lookup paths and an audit trail
A memory object can have an ID, text content, JSON tags, document data, and a vector embedding for similarity search. Swanson also recommends a text index for keyword search, graph links for related entities and memories, temporal fields for validity and last use, and audit events that record who created the memory and how it was used. The record is compact, while an episode can preserve the transcript slice that produced it. That relationship provides provenance and extra context during later recall.
Safe formation starts with redaction and scope checks
Before enrichment or embedding, the system should remove personally identifying information, secrets, and irrelevant material. It can then add metadata, apply agent, team, and organization scope, create an embedding, and link the memory to the episode that produced it. Organization-specific policies can be applied before the record reaches storage. Swanson treats redaction as a first step because sensitive data becomes harder to control after it has entered embeddings or other derived records.
Hybrid retrieval combines evidence instead of trusting one similarity score
Recall is a query against the storage layer. Hybrid search can combine vector similarity, full-text search, relational entity matches, graph traversal, temporal signals, and feedback such as importance or usefulness. The system can fuse results, apply thresholds, re-rank them, and return a context card with scored evidence and provenance. Exact fact lookups may need an ID match rather than semantic similarity. The weights for each search branch can be tuned, and the final result can be limited to the highest-ranked memories.
Feedback determines whether memories should remain, change, or be removed
Memory cannot be a passive repository. It must handle stale facts, bad scope, poisoned instructions, conflicting records, over-recall, irrelevant matches, data leakage, and silent drift as the collection grows. Swanson gives a build-system change as an example of stale memory: advice about NPM can become wrong after a project switches systems. Retrieval outcomes should be recorded as helpful, stale, irrelevant, or harmful. That feedback can change ranking and lifecycle decisions, including promoting useful memories or lowering harmful ones.
"And I feel like if we continue to the logical conclusion of trying to build a memory layer on top of files, we're going to be building our own database."02:43
Who should watch
You are building agents that repeat file reads, queries, or procedures across sessions and need a persistent state model.
Your team is deciding whether markdown files, prompts, or a vector index are enough for memory in a multi-tenant system.
You need to design retrieval and governance together, especially when memories may contain personal data, secrets, conflicting instructions, or stale project practices.
