# Connect AI to Billions of Legal Documents Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora | AI Engineer World's Fair 2026 | 20:36 Source: https://www.youtube.com/watch?v=V-isu4eTHgw Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/connect-ai-to-billions-of-legal-documents Published: 2026-09-16 Tags: latency, search, security ## TL;DR - Legora moved from a shared Elasticsearch cluster to regional clusters, PostgreSQL, and finally one turbopuffer namespace per project as its document corpus grew. - Making the project the storage unit kept inactive projects out of hot query paths and avoided the cache thrashing that pushed P99 latency from 100 milliseconds to 20 seconds. - Turbopuffer uses object storage, memory hierarchies, and namespace-level isolation to support legal search with customer-managed keys and large numbers of cold indexes. ## Summary Jacob Lauritzen explains how Legora built search for collaborative legal work as its corpus grew from hundreds of thousands of documents to billions. The system started with one Elasticsearch cluster, then moved to regional clusters for data residency, and later moved search into PostgreSQL because Legora already operated it. That design failed when hot and inactive projects shared large partitions, causing cache thrashing and raising P99 search and ingestion latency from 100 milliseconds to 20 seconds. Legora moved to turbopuffer, with one namespace per project. Simon Eskildsen explains why this works for regulated customers: namespaces can use separate encryption keys and buckets, while inactive data can remain in object storage. The talk also covers object-storage round trips, memory hierarchies, tree-based vector indexes, and full-text search. The speakers argue that retrieval infrastructure must handle both heavily queried legal material and a long tail of rarely accessed projects. ## Key ideas ### Legora has separate search workloads for projects and legal research [00:34](https://www.youtube.com/watch?v=V-isu4eTHgw&t=34s) Legora supports law firms and in-house legal teams with contract review, contract creation, legal research, and collaboration. Project search stays within one unit of work, which might contain tens of documents or millions. A project could cover an acquisition and include employment agreements, supplier contracts, and other deal documents. Legal research searches across laws, prior cases, and regulations to answer questions or support litigation. These workloads have different shapes, so the infrastructure story begins by separating them. ### Regional processing requirements forced Legora to duplicate its search setup [02:32](https://www.youtube.com/watch?v=V-isu4eTHgw&t=152s) Legora first put all tenants on one Elasticsearch cluster and one blob store. That was simple and worked at smaller scale. Data residency requirements then changed the design. US customers wanted processing in the US, European customers wanted it in the EU, and Australian customers wanted it in Australia. Legora multiplied the setup across regions, which met the requirement but added operational overhead. Large banks and law firms added another demand: their own database and customer-managed encryption keys. ### PostgreSQL failed when hot and inactive projects shared partitions [04:48](https://www.youtube.com/watch?v=V-isu4eTHgw&t=288s) Legora moved search into PostgreSQL because it already used PostgreSQL for OLTP workloads. It used pgvector, specifically DiskANN and TSVector, and partitioned document chunks into about 4,000 partitions. Each project was assigned to partitions by project key. The problem was that inactive projects and active projects landed together. Queries pulled large partitions into memory, then pushed in other partitions and evicted the useful data. Search and ingestion P99 latency rose from 100 milliseconds to 20 seconds. ### One namespace per project separated cold data from active search [06:40](https://www.youtube.com/watch?v=V-isu4eTHgw&t=400s) Legora moved to turbopuffer at roughly 400 million documents and created one namespace per project. This restored BM25 search, improved relevance and latency, and reduced operational work to a single cluster. A namespace that is not being used can stay in object storage instead of occupying the same active partition as a busy project. Legora also benefited from using the blob-based system alongside the raw documents it already stored, with lower cost than the PostgreSQL arrangement. ### Object storage and namespaces provide the isolation model regulated customers need [07:56](https://www.youtube.com/watch?v=V-isu4eTHgw&t=476s) Turbopuffer writes directly to object storage instead of relying on disk replication or Paxos. A namespace is similar to a table or an isolated directory in S3. Each namespace can use a separate encryption key and can live in a separate bucket. Customers can place buckets in their own cloud accounts, re-encrypt data with different keys, and revoke access to a key so the data cannot be decrypted. For Legora, this supports physical and encryption separation across tenants. ### Disabling the SSD cache was simpler than encrypting it [09:58](https://www.youtube.com/watch?v=V-isu4eTHgw&t=598s) Turbopuffer treated its NVMe SSD cache as volatile, like memory, but Legora's customers included the cache in their isolation requirements. The team considered adding encryption to the disk cache. Instead, it disabled the cache and measured the result. Memory caching alone delivered acceptable performance for the affected workloads, so the disk cache stayed off. The experiment avoided adding a special encryption system to one layer of the storage hierarchy. ### Legal research needs a storage layout for hot jurisdictions and cold ones [12:07](https://www.youtube.com/watch?v=V-isu4eTHgw&t=727s) Legora's legal research corpus was approaching 10 billion vectors, with query spikes caused by fan-out. Search must account for jurisdictional hierarchy, temporal validity, overruling decisions, exemptions, and special cases. Some jurisdictions, such as EU law, are queried often, while others are rarely used. Turbopuffer can keep frequently queried namespaces warmer and leave cold jurisdictions in object storage. A few hundred milliseconds to fetch a cold namespace is acceptable for this deep-research workload. ### Tree-based vector indexes reduce object-storage round trips [15:30](https://www.youtube.com/watch?v=V-isu4eTHgw&t=930s) Simon describes Turbopuffer's vector index as a hierarchy of clusters, similar to a complicated B-tree. Graph-based vector search requires traversing nodes, and each step can incur an object-storage round trip. A tree keeps upper levels, which are used on every search, closer to memory while leaves containing the actual cases and documents can remain on SSD or object storage. The design matches the storage hierarchy and avoids pulling the whole corpus into faster, more expensive memory. ### Full-text search can cost more than vector search at web scale [17:45](https://www.youtube.com/watch?v=V-isu4eTHgw&t=1065s) Full-text search stores an inverted index: each token points to the document IDs containing it. A query such as 'New York population' retrieves the relevant token lists, intersects them, and scores documents with BM25. The system must keep round trips and memory bandwidth low while reading dictionaries and large document-ID lists. Common terms create especially large lists, so compression and early elimination matter. Simon says full-text search at web scale is more computationally expensive than vector search. ## Notable quotes - Jacob Lauritzen: "What happened was our latencies would spike. So we went from like search and ingestion P99 of 100 milliseconds into 20 seconds, which you can imagine is a really bad user experience." (06:16) - Simon Eskildsen: "What we did was that we thought we were going to implement encryption into the disc cache but instead we just disabled the disc cache and saw how it fared." (11:10) - Simon Eskildsen: "Everything in Turbopuffer is optimized around doing as much work in as few round trips as possible." (09:19) - Simon Eskildsen: "And counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search." (19:04) ## Tools & references mentioned - Legora - turbopuffer - Elasticsearch - PostgreSQL - pgvector - DiskANN - TSVector - BM25 - Amazon S3 - Google Cloud Storage - Azure Blob Storage - SpaceX - Cursor ## Who should watch - You are building search for a multi-tenant product and need to separate active data from a long tail of inactive projects. - Your customers require data residency, customer-managed encryption keys, or storage isolation at the tenant level. - You are deciding between vector search, full-text search, and a database design that uses object storage as part of its main architecture. ## Related talks - [Scaling Enterprise-Grade RAG: Lessons from Legal Frontier](https://aietalks.com/talks/scaling-enterprise-grade-rag-lessons-from-legal-frontier) (Calvin Qi, Harvey & Chang She, LanceDB, 16:40) - [Realtime Data Connectivity for AI](https://aietalks.com/talks/realtime-data-connectivity-for-ai) (Tanmai Gopal, Hasura, 07:14) - [Building efficient hybrid context query for LLM grounding](https://aietalks.com/talks/building-efficient-hybrid-context-query-for-llm-grounding) (Simrat Hanspal, Hasura, 11:35) - [Turning My Obsidian Vault Into a Local AI Engineer](https://aietalks.com/talks/turning-my-obsidian-vault-into-a-local-ai-engineer) (Filip Makraduli, Superlinked, 47:26) - [OpenRAG: An open-source stack for RAG](https://aietalks.com/talks/openrag-an-open-source-stack-for-rag) (Phil Nash, IBM, 15:52)