# RAG at scale: production-ready GenAI apps with Azure AI Search

Pablo, Microsoft | AI Engineer World's Fair 2024 | 21:53

Source: https://www.youtube.com/watch?v=_2tZaDs-w5s
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/rag-at-scale-production-ready-genai-apps-with-azure-ai-search
Published: 2025-02-13
Tags: embeddings, rag, reranking, search

## TL;DR
- RAG separates the language model's reasoning from domain knowledge stored in an external index.
- Production RAG systems have to handle growing data volume, changing data, query traffic, multi-step workflows, and more file types.
- Azure AI Search combines vector and keyword retrieval, filtering, reranking, quantization, and managed ingestion to support larger RAG applications.

## Summary
Pablo explains why RAG applications become harder when they move from prototypes to production. The basic pattern is simple: an orchestration layer retrieves relevant information, then gives it to a language model with instructions for producing an answer. In practice, systems must handle more users, more data, faster updates, multi-step workflows, and sources beyond plain text. Azure AI Search combines vector search with keyword search, filters, approximate or exhaustive nearest-neighbor search, and a second-stage reranker. Pablo explains why recall-oriented retrieval should reduce a large corpus to a smaller candidate set before a more expensive cross-encoder ranks the results. He also covers increased vector-storage limits, quantization down to one-bit representations, and integrated ingestion for Azure data sources. That pipeline can track changes, process only updates, handle formats such as PDFs, Office files, and images, chunk content, vectorize it, and write it to an index.

## Key ideas
### RAG keeps reasoning in the model and domain facts in an external index
[00:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=54s)
Pablo contrasts three ways to add domain knowledge to a language model: prompt engineering, fine-tuning, and retrieval augmented generation. RAG is useful when the model needs information it did not see during training, such as application data, company information, or user data. The pattern separates reasoning from knowledge. An orchestration component receives a user question, searches a knowledge base for useful passages, and sends candidates plus instructions to the language model. Pablo describes this as mechanically simple, although real applications add tuning and multiple steps.

### Production RAG applications grow along several separate dimensions
[02:57](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=177s)
Pablo says the move from 2023 prototypes to 2024 production applications changes the engineering problem. Once users rely on an application, they ask for more data and faster answers. Data volume grows because successful systems are asked to include more of an organization's information. Data changes more often, query load rises, and retrieval and model calls can appear several times in a single workflow. Applications also bring in more data types and sources. Each pressure affects the retrieval system and the surrounding application, so scaling is more than increasing the size of a vector index.

### Azure AI Search combines vector retrieval with general search features
[05:21](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=321s)
Pablo presents Azure AI Search as a retrieval system that includes vector database capabilities alongside Microsoft's broader search experience. It supports fast approximate nearest-neighbor search and exhaustive search, which can help create recall baselines or inspect retrieval performance. Applications can combine vectors with keyword search, filters, selected fields, ranges, and Boolean expressions. Documents can contain multiple vectors for different content parts or embedding models, and queries can use multiple vectors. The goal is to avoid assembling separate systems for each retrieval requirement.

### Hybrid retrieval can fuse vector and keyword candidates
[07:47](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=467s)
In the notebook demo, Pablo creates an index with categorical, text, and vector fields, then inserts sample records. A vector query returns documents close to a reference vector. Adding search text combines keyword and vector retrieval, and the system fuses and ranks the results. A category filter limits the results to matching metadata. He explains that the requested number of vector candidates is separate from the number of final results: keyword retrieval can contribute additional candidates, after which the combined set is ranked and a selected number is returned.

### A two-stage pipeline uses broad retrieval before expensive reranking
[10:56](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=656s)
Pablo describes quality as a direct consequence of whether retrieval finds the information needed for an answer. Azure AI Search first uses vectors, keywords, and other recall-oriented methods to produce many candidates. A second stage applies a larger ranking model to a smaller set. This model is a cross-encoder that sees the query and document together and predicts how well they correspond. That is more expressive than comparing separate vectors, but it would be impractical to run across an entire large collection. Pablo gives a rough 100-millisecond figure for the reranking model and says the latency often sits behind the language-model call.

### Metadata filters can improve quality before ranking begins
[13:55](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=835s)
Pablo says narrowing the data set with discrete metadata is often the most effective way to improve retrieval quality. If an application knows useful attributes that define the relevant scope, it can filter first and apply ranking methods to the smaller result set. This approach reduces the number of documents competing for the result positions. His example uses categories, but the demo also shows that Azure AI Search supports fuller filter expressions, including combinations, ranges, and other database-style conditions.

### Higher vector density and quantization allow much larger indexes
[16:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=976s)
Pablo describes increased service limits that provide roughly 10 to 12 times the vector density on the same setups, without changing prices. He says the new limits make multi-billion-vector applications possible by provisioning a service and uploading data. Quantization offers another storage tradeoff by replacing full-precision values with narrower types such as int8, or even single-bit representations. He says evaluations retained low-to-mid-90-percent performance for some models with one-bit quantization, and reranking can recover some precision. AI Search can preserve the original vectors and use oversampling, searching compressed vectors before reranking with the stored full-precision data.

### Managed ingestion handles changing files and content beyond text
[20:20](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1220s)
Pablo says teams should not have to build a separate connector and change-tracking process for every data source. Azure AI Search can connect to Azure storage sources such as Blob Storage, OneLake, and Cosmos DB, handle security, track changes, and process only updated content. The ingestion pipeline supports PDFs, Office documents, images, and nested formats. It can chunk content, vectorize it, and write the results to an index. Once configured, the pipeline continues to reflect source-data changes, leaving application developers to focus on retrieval workflows and queries.

## Notable quotes
- "RAG effectively means you separate the reasoning piece of the picture from the knowledge piece of the picture." (01:51)
- "When you go to production you go from, 'Oh, this demo is really cool' to all these users using it at the same time." (03:36)
- "The ranker is these type of rerankers are often called cross encoders." (13:04)
- "You can go from float 32 to one bit, that's 32x the vector density." (18:42)
- "We will deal with all file formats, PDFs, Office documents, images, unpack the nested formats and whatnot, do chunking, do vectorization, and land it on an index all in one go." (20:58)

## Tools & references mentioned
- Azure AI Search
- Microsoft
- OpenAI
- Azure Blob Storage
- OneLake
- Cosmos DB
- BM25
- HNSW
- Ada

## Who should watch
- You are moving a RAG prototype into production and need to reason about data volume, query traffic, update frequency, or multi-step retrieval workflows.
- Your current system combines separate vector, keyword, filtering, reranking, and ingestion components and you want to see how Azure AI Search brings those functions together.
- You need to store more vectors or index PDFs, Office files, images, and changing Azure data sources without writing every ingestion connector yourself.

## Related talks

- [RAG in 2025: State of the Art and the Road Forward](https://aietalks.com/talks/rag-in-2025-state-of-the-art-and-the-road-forward) (Tengyu Ma, MongoDB, 18:48)
- [RAG for VPs of AI](https://aietalks.com/talks/rag-for-vps-of-ai) (Jerry Liu, LlamaIndex, 26:51)
- [Building Production-Ready RAG Applications](https://aietalks.com/talks/building-production-ready-rag-applications) (Jerry Liu, LlamaIndex, 18:35)
- [Build, Evaluate and Deploy a RAG-Based Retail Copilot with Azure AI](https://aietalks.com/talks/build-evaluate-and-deploy-a-rag-based-retail-copilot-with-azure-ai) (Cedric Vidal, David Smith & Miguel Martinez, Microsoft, 1:57:58)
- [RAG Agents in Prod: 10 Lessons We Learned](https://aietalks.com/talks/rag-agents-in-prod-10-lessons-we-learned) (Douwe Kiela, Contextual AI, 16:56)