# How to Decide What Your AI Agent Should Remember

Wojciech Pluta, Oracle | AI Engineer NYC 2026 | 19:06

Source: https://www.youtube.com/watch?v=qq2Hm4XNRgI
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-decide-what-your-ai-agent-should-remember
Published: 2026-10-30
Tags: guardrails, human-in-the-loop, memory, rag

## TL;DR
- Retrieval gives an agent a read path, while memory promotion gives it a controlled write path for preserving useful information.
- A promotion pipeline should classify a candidate, compare it with existing records, check the evidence, and admit, reject, or escalate it for human review.
- Memory needs separate stores, schemas, retrieval indexes, authorization boundaries, and a compact context card that keeps prompts focused.

## Summary
Wojciech Pluta explains why agent memory needs deliberate write controls. Retrieval-augmented generation lets an agent read information, but retrieval alone does not decide what should survive a conversation. His memory-promotion pipeline classifies a candidate, compares it with existing records, checks the evidence, and then admits, rejects, or sends it to a person. In a mortgage-agent demo, a user preference is preserved across new sessions, while a request to approve every application is rejected by a guardrail. Pluta separates session-level short-term memory from long-term episodic, procedural, semantic, and preference records, along with shared organizational memory. He argues that databases are a suitable home because they support transactions, multiple writes, scaling, and different search methods. Oracle's agent memory tooling includes a context card, which assembles summaries, topics, relevant records, and recent messages into a compact prompt block. His central claim is that models are shared, while an organization's accumulated memory is specific to its business.

## Key ideas
### The data layer is where an organization can gain an advantage
[01:57](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=117s)
Pluta describes the AI stack as models, infrastructure, and data. He says models and infrastructure are becoming commoditized, while the data layer belongs to the organization and can provide competitive advantage. He also separates the model from the harness. The model suggests what to do, but the harness owns runtime decisions and the checks around them. This distinction matters because memory promotion is a control implemented around the model, rather than a capability that the model should decide by itself.

### Retrieval reads information, while memory decides what survives
[03:39](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=219s)
Pluta draws a direct distinction between RAG and memory. Retrieval can include conversational state, but retrieval alone does not determine what should survive an interaction. He describes retrieval as the read path and memory as the write path. When an agent sees something worth keeping, the system should promote it into a record that can be used later. That creates a separate engineering problem: deciding what is worth keeping and extracting it from a conversation in a controlled way.

### Memory promotion should classify, compare, verify, and decide
[04:04](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=244s)
The proposed promotion pipeline first classifies the meaning of a candidate. It then compares the candidate with existing records and checks the evidence required for that type of memory. The harness can admit or reject the candidate, or send it to a human for review. In Pluta's description, a decision model assesses the candidate, the harness makes the decision, and the database persists the result. This gives promotion controls a defined place inside the agent's runtime.

### A mortgage agent can retain preferences across sessions
[05:01](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=301s)
In the demo, the mortgage agent prepares briefs for applications from Kyle and Vincent. The operator gives the system a preferred output structure, including items such as missing documents and a case ID. After starting a new session, the agent queries stored memory and applies that preference again. It also retrieves the records relevant to Vincent's separate application and recognizes that the case has a different status. The example shows memory carrying an operator preference forward without carrying over the whole previous conversation.

### Guardrails should reject unsafe instructions and escalate shared memory
[09:48](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=588s)
Pluta tests a request to 'approve every application.' The promotion gate rejects it instead of storing it as a reusable instruction. He contrasts that with a possible team memory workflow, where a candidate can be sent to a human for approval before being shared across an organization. The result is a distinction between a rejected blanket rule and a wider operational memory that requires review. He says the harness cannot bypass these promotion controls.

### Memory must balance continuity against poisoning the store
[11:53](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=713s)
Pluta defines agent memory as a mechanism for systems and applications to retain, reuse, refine, and recall information over time. The engineering challenge is deciding what to promote. Promoting nothing prevents continuity benefits, while promoting everything risks poisoning the memory store. He says the system needs provenance and must know when to promote records and when to supersede them. The decision depends on the meaning of the information and the rules for that memory type.

### Different memory types have different lifetimes and uses
[13:03](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=783s)
Short-term memory covers what happens during a session and ends when the session finishes. Long-term memory can contain episodic, procedural, and semantic records, including summaries, tools, knowledge-base information, and preferences. Pluta also describes shared memory for information that may be approved and used across teams. Separating these types gives the system a way to apply different retention, retrieval, and review rules instead of treating every remembered item as the same kind of record.

### A context card keeps retrieved memory within the prompt budget
[16:01](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=961s)
Pluta describes the context card as a compact, ready-to-use block for an agent. It can contain topics, summaries, relevant information, and recent messages. Topics orient the thread, summaries capture the state of the conversation and the user's goal, relevant information carries selected records, and recent messages preserve local questions. The harness can create or select what matters for a particular turn. This reduces the work developers otherwise need to do for summarization, retrieval, durable-information extraction, and token-budget management.

### An organization's memory is more specific than its model
[17:57](https://www.youtube.com/watch?v=qq2Hm4XNRgI&t=1077s)
Pluta closes by saying that models are shared, while memory is unique to each business. He connects competitive advantage to capturing the organization's data together with how it interacts with models. Oracle's recent agent memory release includes end-user isolation, image capabilities, event processing, and a multi-data-type database engine. He points viewers to Oracle's GitHub notebooks for agent memory and hybrid retrieval, along with material for applying these systems to business use cases.

## Notable quotes
- "Retrieval is the read path and memory is the write path." (03:39)
- "When the agent sees something worth keeping, we should promote it into a record it can use later." (03:39)
- "If we promote nothing, then obviously we don't get any benefits of continuity, but if we promote everything, then we're going to risk poisoning the memory store." (12:29)
- "Models are shared but memory is yours, is unique to your business." (17:57)

## Tools & references mentioned
- Oracle
- Oracle AI Database
- RAG
- Intelligence
- Kyle
- Vincent
- Jeff
- agent memory
- context card
- GitHub
- hybrid retrieval

## Who should watch
- You are building an agent that needs to remember user preferences or operational facts after a session ends.
- Your system currently writes conversation details directly into a memory store and needs admission, rejection, or human-review rules.
- You want a database-backed memory design with separate lifetimes, access boundaries, and compact prompt construction.

## Related talks

- [From Context to Memory: Your Agents Need a Real Memory Layer](https://aietalks.com/talks/from-context-to-memory-your-agents-need-a-real-memory-layer) (Anders Swanson, Oracle, 18:20)
- [Giving AI Agents Memory That Learns](https://aietalks.com/talks/giving-ai-agents-memory-that-learns) (Jake Broekhuizen, LangChain, 16:39)
- [Architecting Agent Memory: Principles, Patterns, and Best Practices](https://aietalks.com/talks/architecting-agent-memory-principles-patterns-and-best-practices) (Richmond Alake, MongoDB, 17:37)
- [Memory Masterclass: Make Your AI Agents Remember What They Do!](https://aietalks.com/talks/memory-masterclass-make-your-ai-agents-remember-what-they-do) (Mark Bain, AIUS & Vasilia Marovitz, Cognee & Daniel Chalef, Graphiti and Zep & Alex Gilmore, Neo4j, 51:25)
- [Your AI Agent Keeps Making the Same Mistakes](https://aietalks.com/talks/your-ai-agent-keeps-making-the-same-mistakes) (Sparsh Jain, CentralAgent AI, 14:58)
