AI-powered search combines traditional keyword retrieval with related concepts, user intent, and some reasoning.
2
Good retrieval systems need more than embeddings. Structured filters, domain-specific models, reranking, query decomposition, and evaluation all matter.
3
Search is moving toward multimodal embeddings, instruction-guided retrieval, and databases that combine embedding, reranking, and query operations in one platform.
Summary
Frank Liu defines AI-powered search as a system that can find related concepts without identical wording, understand a user's intent, and follow some instructions. He explains how retrieval supports grounded generation, then uses three applications to describe practical lessons. Code search needs evaluation because no single embedding model fits every use case. Legal and other domain-specific search often needs structured filters alongside embeddings. Agentic retrieval can expand or decompose a query and search several sources before producing an answer. Liu then describes where search is heading. Multimodal models will place text and images, and later audio, into shared semantic spaces. Instruction tuning will let users steer retrieval toward a particular aspect of a query. He also expects databases to combine embeddings, reranking, query augmentation, and decomposition in one platform, reducing the number of separate components engineers must assemble.
AI-powered search finds related meaning and user intent
Liu defines AI-powered search as a system that can find related concepts even when the wording is different. It goes beyond TF-IDF and BM25 by combining keyword retrieval with semantic similarity. He gives the example of a user searching for something because their best friend is sick. A useful system might recommend a get-well basket, even though the query does not contain those exact product words. Liu also says search systems should understand intent, perform some reasoning, and follow instructions. These abilities make retrieval useful for applications where the user's goal is more important than literal word matching.
Liu presents retrieval-augmented generation as a common use of AI-powered search. Without retrieval, a language model may hallucinate, refuse to answer, or produce a generic response. With retrieval, the system searches relevant source material and passes it to the language model, which can then produce a more grounded answer. The basic flow is to embed unstructured data such as text, PDFs, Word documents, Google Drive files, and PowerPoint presentations. A query is embedded into the same space, and the system retrieves the most relevant documents. Liu calls embedding quality a core part of search and retrieval systems.
Embedding models must be evaluated for each application
For chatting with a codebase, Liu describes a classic RAG system with reranking. He uses Continue.dev as an example and says its evaluation found Voyage Code 3 performed best for that application. Code search has particular requirements because the model needs to understand source code, documentation, and developers' terminology. Liu's broader lesson is that there is no one-size-fits-all embedding model or language model. Teams should evaluate candidate models against their own application rather than assuming that a generally strong model will produce the best retrieval results for their data.
Structured data belongs inside the retrieval system
Liu says embeddings alone are often insufficient for a powerful search system. A domain such as legal search may need semantic retrieval over documents, along with filters for the state, whether a document is official, or other specific attributes. He gives Voyage Law 2 as an example of a domain-specific embedding model, but says the structured information still needs to be included. Filtering can happen directly in the vector store or after retrieval. The search system therefore needs to combine semantic similarity with metadata and other structured sources that constrain the result set.
Agentic retrieval can expand and decompose queries
Liu describes agentic retrieval as a feedback loop rather than a simple input-output operation. After an initial query and search, a language model may expand the query or break it into smaller searches. For example, a request for X's 2024 earnings could be decomposed into separate searches for Q1, Q2, Q3, and Q4 earnings. Those queries can then be sent to different vector stores or databases. Liu connects this pattern to conversational applications, where agents need search systems and embeddings that can work with conversational data.
Multimodal embeddings will put different data types in one space
Liu says the future of search and retrieval is multimodal. He means foundational models that can understand images and text together, and eventually combinations such as images, text, and audio. He uses Voyage Multimodal 3 as an example of a model that can embed text, images, or interleaved text and images into a shared semantic space. In his example, a query about strong language models retrieves a Claude 3.5 blog post. The same approach can let search systems compare and retrieve across data types instead of treating images and text as separate collections.
Instructions can steer the vector produced for a query
Liu expects embedding models to move beyond taking a query or document and returning an unqualified vector. Users will be able to add an instruction that steers retrieval in a particular direction. For example, a search could ask for documents that examine one specific aspect in detail. The instruction would shape the representation used for retrieval, rather than leaving the embedding model to interpret only the original query. Liu links this development to instruction tuning and, to a secondary extent, reasoning in search systems.
A single database could combine the retrieval pipeline
Liu describes an agent-native database as a platform that brings several retrieval functions together. Current systems often require multiple components for embeddings, vector search, reranking, query augmentation, and query decomposition. He sees MongoDB's combination with Voyage AI as a step toward a single data platform that can handle those operations. The database could embed data, rerank results, and perform query changes within one piece of infrastructure. Liu expects this type of platform to become more common during 2025 and 2026.
"The capability to move directly to something that is just a single piece of infrastructure that does the embedding for you."11:40
Who should watch
You are building a RAG or semantic search system and need practical guidance on model selection, filters, and reranking.
Your application searches code, legal documents, or another specialized domain where generic embeddings may perform poorly.
You are designing an agent that must decompose queries, work with conversational data, or retrieve across text and images.
