Stop Using RAG as Memory

Daniel Chalef, Zep07:02 · Jul 2025 · 2,474 views
Thumbnail for Stop Using RAG as Memory Watch on YouTube
TL;DR
  1. 1

    Memory should be modeled around the business domain instead of storing arbitrary facts in a vector database.

  2. 2

    Semantic similarity can retrieve irrelevant facts because it does not capture business relevance, causality, or relationships.

  3. 3

    Zep uses developer-defined entities, fields, and business rules in a temporal knowledge graph so agents can retrieve focused business context.

Summary

Daniel Chalef argues that vector search is a poor foundation for agent memory. It retrieves facts that are semantically similar to a query, even when those facts have no relevance to the application. In his media assistant example, a request about favorite tunes can retrieve the user's dog, Melody, because the name resembles a musical term. Chalef says developers should model memory around their business domain. His finance coach demo uses Zep and Graphiti to store explicit objects such as financial goals, debts, and income sources. Developers define entity types, fields, descriptions, and business rules with tools such as Pydantic, Zod, or Go structs. The application can then run filtered searches for a financial snapshot, rather than retrieving any vaguely related fact. A new rent value is added to the graph and appears on the user's knowledge graph with the developer-defined structure.

Key ideas
00:00

Agent memory needs a business domain model

Daniel Chalef opens by saying there is no one-size-fits-all memory. The memory model should match the business domain. In the Graphiti framework, developers can create custom entities and edges for their particular application. These entities and relationships correspond to business objects, so the application can represent the information it needs instead of treating every conversational fact as equally useful.

01:05

Irrelevant facts pollute conversational memory

Chalef uses ChatGPT and a media assistant to show how arbitrary memory goes wrong. A media assistant should remember preferences about jazz, NPR, podcasts, and The Daily. Instead, it may also save facts such as the user's 7 a.m. wake-up time or the name of the user's dog, Melody. Those details do not belong to the media-player domain. When irrelevant facts are stored and later retrieved, the agent can produce inaccurate answers or hallucinations.

02:13

Semantic similarity does not equal business relevance

Many agent frameworks generate facts and put them into a vector database such as Redis. Retrieval then returns facts that are semantically similar to the query. For a request about favorite tunes, Melody may be returned because Melody is also the name of a dog. Chalef says this is the technical limitation of vector representations: they are projections into an embedding space and do not provide causal or relational links. Better semantic search does not solve domain-aware memory.

03:56

Developers should define explicit business objects

In the finance coach demo, the application asks about buying a house, annual income, and student loan debt. The memory shown in Zep contains explicit business objects such as financial goals, debts, and income sources. Developers define these objects and their fields, rather than allowing the system to collect any fact from the conversation. Chalef presents this as a way to make stored memory fit the finance application.

04:32

Schemas let applications enforce business rules

Chalef shows a TypeScript financial goal schema built with Zep's SDK. An entity type can have a description and fields with defined values and business rules. The same approach can be used with Pydantic, Zod, or Go structs. When the application starts, it registers the objects with Zep so the graph knows the ontology that the developer wants to use.

05:24

Retrieval can be narrowed to the objects an agent needs

The finance coach includes a tool for retrieving a financial snapshot. That tool runs several Zep searches concurrently and filters the results by specific node types. This gives the agent a focused view of the user's financial state instead of a broad collection of semantically related memories. The developer controls which types of objects can be returned for the task.

05:50

New facts become structured graph data

When Chalef adds a $5,000 monthly rent value, Zep processes the message and captures it within seconds. The front end then shows the user's knowledge graph, including a debt account entity and the fields defined by the developer. The graph view makes the stored structure visible and lets the application retrieve information through the defined entity model.

"We need domain-aware memory, not better semantic search."03:26
Who should watch
  • You are building an agent that stores conversational facts and keeps retrieving unrelated details.
  • Your application has business objects such as debts, goals, accounts, or preferences that need structured fields and filtering.
  • You are deciding whether vector search alone is sufficient for an agent's long-lived memory.