Legal RAG has to handle massive document collections, complex expert queries, sensitive data, and strict retrieval-quality requirements.
2
Harvey evaluates retrieval with a range of methods, from expert review to automated precision and recall checks, because no single evaluation covers the whole system.
3
LanceDB combines vector search, full-text search, reranking, multimodal storage, analytics, and training workflows on an object-store-backed architecture.
Summary
Calvin Qi describes Harvey's retrieval problems across uploaded documents, project vaults, and global legal corpora. The data can reach tens of millions of large documents, while queries may combine semantic meaning, dates, regulation identifiers, multiple legal references, filters, and domain jargon. Harvey spends substantial effort on evaluation, using expert reviews, curated criteria, and automated retrieval metrics to support faster iteration. Chang She presents LanceDB as an AI-native multimodal lakehouse built on the open source Lance format. It stores text, embeddings, images, video, audio, tabular data, and time series together, while supporting online search, offline processing, analytics, and training. The speakers argue that domain experts, flexible infrastructure, and fast evaluation cycles matter more than relying on a single retrieval technique or agent pattern.
Harvey's retrieval workloads range from individual uploads to global legal corpora
Harvey supports several data scales. Its assistant handles on-demand uploads, roughly in the 1 to 50 range. Vaults collect the contracts, litigation documents, and emails for a large deal or data room. Data corpuses provide knowledge bases for legislation, case law, taxes, and regulations in particular countries. These workloads differ in size and organization, so the same retrieval design cannot be applied everywhere. The largest corpuses can contain tens of millions of documents, and individual documents are often long and dense.
Legal queries combine semantic meaning with filters, identifiers, and domain language
Qi uses a covered-bonds question to show why legal retrieval is difficult. The query asks which regime applies to bonds issued before a particular date, while naming an EU directive and a specific article. A system must understand the meaning of the question, apply an implicit date filter, search a specialized EU legal collection, match exact regulation identifiers, connect multiple legal references, and interpret domain abbreviations. Harvey therefore needs to break a query into parts and use the retrieval method that fits each part.
Evaluation has to cover both expensive expert judgment and fast automated checks
Qi says Harvey spends more time validating systems than applying fancy algorithms or agent techniques. There is no single evaluation that captures every failure. At the high-fidelity end, legal experts review outputs and write reports, which is expensive but detailed. Curated criteria can support synthetic or automated checks at a lower cost. Faster tests measure retrieval precision and recall or check deterministic conditions, such as whether results came from the right folder, section, or keyword set. Together, these evaluations provide signals at different speeds and levels of detail.
Domain experts help turn legal structure into searchable data
Harvey works with domain experts to organize datasets by country, legal category, and other properties. The team uses expert guidance to create heuristics and LLM processing techniques where automation is practical. This work affects how documents are represented, indexed, queried, and pre-processed. The data is also subject to customer-specific privacy and retention rules. Some material must be stored separately, while other documents can only be retained for a limited period because of legal requirements.
The retrieval infrastructure must support online serving and offline experimentation
The system needs low-latency online queries and efficient offline work such as ingestion, reingestion, and machine-learning experiments. Harvey wants data and ML teams to spend their time on application behavior and retrieval quality instead of manually tuning databases or scaling infrastructure. At the same time, the platform needs telemetry, flexible query patterns, exact matching, semantic search, filters, sorting, and dynamic navigation. Full automation is not enough by itself, so teams still need awareness of how the infrastructure behaves.
LanceDB combines retrieval with a broader multimodal data platform
She describes LanceDB as an AI-native multimodal lakehouse rather than only a vector database. Data can remain in object storage while the same foundation supports search, retrieval, analytics, training, and pre-processing. LanceDB can combine multiple vector columns with vector and full-text search, then apply reranking through a Python or TypeScript API. Its distributed architecture separates cloud compute, memory, and storage, allowing the system to support both offline workloads and online serving.
The Lance format keeps large multimodal data and derived features together
The open source Lance format is designed for AI data that includes large blobs alongside scalar fields and embeddings. She contrasts it with storage setups that lack efficient random access, large-blob support, or convenient schema evolution. Lance provides fast random access for search and shuffle, fast scans for analytics and training, and storage for mixed blob and scalar data. It also works with Apache Arrow and can connect to Spark, Ray, PyTorch, pandas, and Polars, reducing the need to keep separate copies synchronized.
RAG teams should optimize for flexibility because the technology keeps changing
Qi's final advice is to study the data and query patterns with domain experts, then build for rapid iteration. Models, tools, context windows, and application patterns are changing, so the system should not depend on fixed assumptions. Good evaluation sets and automated procedures make it easier to tell whether a change improves accuracy. She adds that data infrastructure must account for multimodal inputs, heavier vector and embedding workloads, diverse queries, and continued growth in the amount of public and private data teams want to ingest.
"I'd say investing in eval driven development is a huge huge key to building these systems and making sure they're good, especially when it's a tough domain that you don't inherently know much about as maybe an engineer or a researcher."04:54
Who should watch
You are building RAG for legal, tax, compliance, or another expert domain where queries combine exact references with natural-language meaning.
Your retrieval system handles large document collections and you need an evaluation process that supports both expert review and quick automated iteration.
You are deciding whether vector search infrastructure should also support full-text search, multimodal data, analytics, ingestion, and model training.
