# Stop Using RAG as Memory

Daniel Chalef, Zep | AI Engineer World's Fair 2025 | 07:02

Source: https://www.youtube.com/watch?v=T5IMo5ntyhA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/stop-using-rag-as-memory
Published: 2025-07-22
Tags: memory, rag, structured-outputs

## TL;DR
- Memory should be modeled around the business domain instead of storing arbitrary facts in a vector database.
- Semantic similarity can retrieve irrelevant facts because it does not capture business relevance, causality, or relationships.
- Zep uses developer-defined entities, fields, and business rules in a temporal knowledge graph so agents can retrieve focused business context.

## Summary
Daniel Chalef argues that vector search is a poor foundation for agent memory. It retrieves facts that are semantically similar to a query, even when those facts have no relevance to the application. In his media assistant example, a request about favorite tunes can retrieve the user's dog, Melody, because the name resembles a musical term. Chalef says developers should model memory around their business domain. His finance coach demo uses Zep and Graphiti to store explicit objects such as financial goals, debts, and income sources. Developers define entity types, fields, descriptions, and business rules with tools such as Pydantic, Zod, or Go structs. The application can then run filtered searches for a financial snapshot, rather than retrieving any vaguely related fact. A new rent value is added to the graph and appears on the user's knowledge graph with the developer-defined structure.

## Key ideas
### Agent memory needs a business domain model
[00:00](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=0s)
Daniel Chalef opens by saying there is no one-size-fits-all memory. The memory model should match the business domain. In the Graphiti framework, developers can create custom entities and edges for their particular application. These entities and relationships correspond to business objects, so the application can represent the information it needs instead of treating every conversational fact as equally useful.

### Irrelevant facts pollute conversational memory
[01:05](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=65s)
Chalef uses ChatGPT and a media assistant to show how arbitrary memory goes wrong. A media assistant should remember preferences about jazz, NPR, podcasts, and The Daily. Instead, it may also save facts such as the user's 7 a.m. wake-up time or the name of the user's dog, Melody. Those details do not belong to the media-player domain. When irrelevant facts are stored and later retrieved, the agent can produce inaccurate answers or hallucinations.

### Semantic similarity does not equal business relevance
[02:13](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=133s)
Many agent frameworks generate facts and put them into a vector database such as Redis. Retrieval then returns facts that are semantically similar to the query. For a request about favorite tunes, Melody may be returned because Melody is also the name of a dog. Chalef says this is the technical limitation of vector representations: they are projections into an embedding space and do not provide causal or relational links. Better semantic search does not solve domain-aware memory.

### Developers should define explicit business objects
[03:56](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=236s)
In the finance coach demo, the application asks about buying a house, annual income, and student loan debt. The memory shown in Zep contains explicit business objects such as financial goals, debts, and income sources. Developers define these objects and their fields, rather than allowing the system to collect any fact from the conversation. Chalef presents this as a way to make stored memory fit the finance application.

### Schemas let applications enforce business rules
[04:32](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=272s)
Chalef shows a TypeScript financial goal schema built with Zep's SDK. An entity type can have a description and fields with defined values and business rules. The same approach can be used with Pydantic, Zod, or Go structs. When the application starts, it registers the objects with Zep so the graph knows the ontology that the developer wants to use.

### Retrieval can be narrowed to the objects an agent needs
[05:24](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=324s)
The finance coach includes a tool for retrieving a financial snapshot. That tool runs several Zep searches concurrently and filters the results by specific node types. This gives the agent a focused view of the user's financial state instead of a broad collection of semantically related memories. The developer controls which types of objects can be returned for the task.

### New facts become structured graph data
[05:50](https://www.youtube.com/watch?v=T5IMo5ntyhA&t=350s)
When Chalef adds a $5,000 monthly rent value, Zep processes the message and captures it within seconds. The front end then shows the user's knowledge graph, including a debt account entity and the fields defined by the developer. The graph view makes the stored structure visible and lets the application retrieve information through the defined entity model.

## Notable quotes
- "There's no one-size-fits-all memory." (00:00)
- "Irrelevant facts pollute memory." (02:13)
- "Semantic similarity is not business relevance." (03:26)
- "We need domain-aware memory, not better semantic search." (03:26)

## Tools & references mentioned
- Zep
- Graphiti
- ChatGPT
- Redis
- Python
- TypeScript
- Go
- Pydantic
- Zod

## Who should watch
- You are building an agent that stores conversational facts and keeps retrieving unrelated details.
- Your application has business objects such as debts, goals, accounts, or preferences that need structured fields and filtering.
- You are deciding whether vector search alone is sufficient for an agent's long-lived memory.

## Related talks

- [GraphRAG](https://aietalks.com/talks/graphrag) (Mitesh, NVIDIA & Ching Kyong Lamb, PO.AI & Daniel Chalev, Zep & Mark Bane & Vasilia Marovitz, Cognify & Alex Gilmore, Neo4j & Ola Mabad, Cisco & Michael Hunger & Steven Shen, Neo4j & Tom Smoker, YAL.ai, 4:08:05)
- [CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens](https://aietalks.com/talks/crabrag-why-automated-assistants-need-graph-memory-not-more-tokens) (Stephen Chin, Neo4j, 20:42)
- [Agentic GraphRAG: Simplifying Retrieval Across Structured and Unstructured Data](https://aietalks.com/talks/agentic-graphrag-simplifying-retrieval-across-structured-and-unstructured-data) (Zach Blumenfeld, Neo4j, 15:25)
- [Architecting Agent Memory: Principles, Patterns, and Best Practices](https://aietalks.com/talks/architecting-agent-memory-principles-patterns-and-best-practices) (Richmond Alake, MongoDB, 17:37)
- [Memory Masterclass: Make Your AI Agents Remember What They Do!](https://aietalks.com/talks/memory-masterclass-make-your-ai-agents-remember-what-they-do) (Mark Bain, AIUS & Vasilia Marovitz, Cognee & Daniel Ch⁠alef, Graphiti and Zep & Alex Gilmore, Neo4j, 51:25)