Simple vector search is enough for basic RAG, but more capable applications need retrieval systems that can update from people, agents, and interactions with the world.
2
Retrieval quality depends on the embedding model, chunking strategy, and the ability to exclude irrelevant results, because distractors in the context can sharply reduce an application's performance.
3
Chroma is building a data layer for AI applications with relevance feedback, scalable deployment, hybrid search, multimodal data, and model selection.
Summary
Anton Troynikov describes basic RAG as an open loop: embed a document corpus, retrieve nearest neighbors for a query, and place the results in an LLM context window. He argues that agents and other advanced applications need retrieval that can learn from human feedback, update itself, and store an agent's interactions with the world. He then covers unresolved retrieval problems, including choosing an embedding model, chunking documents, and deciding whether a result is relevant at all. Returning a fixed number of nearest neighbors can add irrelevant context, such as bird articles for a question about fish. Troynikov discusses relevance feedback, reranking, keyword search, metadata filters, and lightweight models that can estimate relevance. He also outlines Chroma's plans for distributed deployments, hybrid installations, multimodal retrieval, and model selection.
Troynikov presents basic RAG as an open loop and says powerful applications need more than a static vector index. Human feedback can adapt retrieval to a specific task, model, user, and data set. Agents also need to update their own memory, while world-model applications must store interactions and change their working data based on what happened. That creates a dynamically changing data set. A search index built only for fixed queries cannot support these capabilities by itself.
Agents can learn and recall skills through retrieval
Troynikov points to the Voyager Minecraft research as an early example of a more capable retrieval system. The agent learns skills in an environment, recognizes when it encounters a similar situation, and recalls the relevant skill. Some of the more complex skills were learned from human demonstrations and then added to the retrieval system. His point is that future AI applications may need memory that stores learned procedures and retrieves them when the current situation matches.
AI retrieval must return relevant information without distractors
For AI applications, retrieving relevant information is only half the problem. The system also needs to avoid irrelevant information. Troynikov says empirical research shows that distractors in the model context can make the whole application perform much worse. This creates practical questions about which embedding model works for a particular data set, how documents should be chunked, and how to determine whether a retrieved result is relevant to the user's task.
Production data is needed to evaluate retrieval choices
Troynikov says existing academic retrieval benchmarks appear mostly saturated because they are synthetic and may not reflect how AI applications use retrieval. Developers can apply open-source benchmark tooling to their own data and use relevance feedback from production. They can construct data sets based on what matters in their application, then compare embedding models against those examples. He also mentions vendor evaluation tools, while saying it is not yet clear which approach is best.
Embedding spaces may be transferable between models
Embedding models trained with similar objectives and roughly similar data tend to learn representations that differ by an affine linear transform, according to the research Troynikov discusses. That suggests one embedding space might be projected into another with a simple learned transformation. If this works well on an application's own data, choosing a particular embedding model may matter less than expected. The approach still depends on finding the transformation from relevant application data.
Chunking has to preserve the right semantic boundaries
Chunking exists partly because LLM context windows have limited length, but the chunks also need to preserve the content that retrieval should find. Troynikov says chunking should respect the natural structure of human-produced data, since that structure often signals semantic boundaries. He mentions tools such as NLTK, LangChain, and LlamaIndex. Chroma is also exploring lightweight language models that detect likely boundaries from next-token perplexity, along with embedding continuity methods that look for discontinuities across sliding windows.
A fixed nearest-neighbor count can produce no useful answer
Troynikov gives a concrete failure case: a corpus containing every English Wikipedia page about birds receives a query about fish. A system that always returns the five nearest neighbors will return five documents, even though none may be relevant. Possible responses include human relevance feedback, auxiliary reranking based on user behavior, keyword search, and metadata filters. His preferred research direction is a conditional relevance signal that depends on the data, task, user, model, and individual query.
Chroma is expanding from a local store into an AI data layer
Troynikov says Chroma is building a horizontally scalable cluster version after its single-node version. He describes a database service technical preview, hybrid deployments for enterprise clusters, multimodal data support, and model selection. Image and voice understanding will allow retrieval systems to work with more than text, which introduces questions such as aesthetic quality as well as relevance. Chroma's stated goal is to handle data-layer concerns so application developers can focus on application logic.
"You need a lot more than simple Vector search to build some of the more powerful more promising applications that take rag in the future."01:11
Who should watch
You are building a RAG application and need to evaluate embeddings, chunking, or relevance on your own production data.
Your agent needs memory that can change from human feedback or from the agent's interactions with an environment.
You are deciding whether a vector store needs keyword search, metadata filters, reranking, or multimodal support.
