# The Hidden Costs of Building Your Own RAG Stack

Ofer, Vectara | AI Engineer Summit 2025 | 15:14

Source: https://www.youtube.com/watch?v=62U6FLUCPWs
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-hidden-costs-of-building-your-own-rag-stack
Published: 2025-02-22
Tags: observability, rag, reliability, security

## TL;DR
- An enterprise RAG system requires much more than a vector database and an LLM, including parsing, chunking, retrieval, evaluation, access control, and monitoring.
- DIY RAG stacks can develop latency, scaling, cost, security, vendor coordination, staffing, and language-support problems as they move into production.
- A turnkey RAG platform hides and maintains these components while providing APIs for indexing and querying, with deployment options in a customer's VPC or on premises.

## Summary
Ofer explains why a RAG proof of concept can become difficult to operate at enterprise scale. A production system must ingest varied sources, parse documents and tables, chunk text, create embeddings, search with vectors and keywords, rerank results, generate answers, detect hallucinations, and preserve citations. Each part adds work and can affect latency, cost, quality, security, and support. He describes seven problem areas: response quality, latency, scaling and cost, security and compliance, vendor coordination, staffing and maintenance, and non-English language support. Vectara's alternative is a managed RAG platform with APIs for indexing and querying, plus deployment in a customer's VPC or on premises. Ofer also describes Vectara's hallucination evaluation model, access controls, prompt attack prevention, and observability. His argument is practical: teams should account for the ongoing engineering and operational work behind a RAG stack, rather than judging the effort by the first working demo.

## Key ideas
### A RAG system has separate ingestion and query flows
[00:20](https://www.youtube.com/watch?v=62U6FLUCPWs&t=20s)
Ofer defines RAG as a way to ground an LLM in a company's own data. During ingestion, the system extracts content from documents or databases, chunks text into smaller pieces, creates embeddings, and stores those embeddings in a vector database. During a query, the question is embedded and matched against the stored vectors. Production systems often add hybrid search, which combines vector and keyword search, plus reranking based on relevance, diversity, or business rules. The retrieved chunks are sent to the LLM with a prompt. The system may then check for hallucinations and preserve the source path so the answer can include citations.

### Response quality requires ongoing work after the first demo
[05:13](https://www.youtube.com/watch?v=62U6FLUCPWs&t=313s)
Building a high-quality RAG system requires more than selecting an embedding model and connecting a vector store. Ofer says teams must invest in parsing, chunking strategies, table parsing and table understanding, hybrid search, and other retrieval components. They also need continuous evaluation as data is added or refreshed and as system components change. The target is for answer quality to remain the same or improve over time. Hallucination prevention and correction add another layer of complexity. Explainability also needs engineering support, because the application must retain where retrieved facts came from so users can inspect the source.

### Multiple retrieval components can make latency difficult to control
[06:26](https://www.youtube.com/watch?v=62U6FLUCPWs&t=386s)
Ofer says latency often rises because a query passes through several components that are not fully coordinated. A simple vector search may be fast, while hybrid search and additional rankers add more processing. The total flow is constrained by its slowest component. If one service has a latency problem, the complete query experiences that delay. This makes production latency harder to predict than the initial prototype suggests. Users expect answers quickly, so the system needs more than a fast vector database. The whole retrieval and generation path has to work within the required response time.

### Production scale raises both infrastructure and token costs
[07:26](https://www.youtube.com/watch?v=62U6FLUCPWs&t=446s)
A production RAG stack can include embedding models, LLMs, vector databases, document-parsing services, table-understanding tools, and other supporting components. Ofer says maintaining low latency and good quality requires capable versions of these components, which brings costs for GPUs, CPUs, storage, and external services. The cost profile changes as a system grows from a small collection of documents to thousands, hundreds of thousands, or millions. Token costs also become harder to maintain. The concern is therefore the combined operating cost of the full stack, rather than the price of any single model or database.

### Access controls and sensitive data must follow the RAG flow
[08:09](https://www.youtube.com/watch?v=62U6FLUCPWs&t=489s)
Ofer focuses on attribute-based access control as a common enterprise requirement. If documents are visible only to a particular department or executive, the RAG system must apply the same restrictions when it retrieves and presents information. Sensitive data such as PHI and PII must also be ingested and handled correctly throughout the stack. A failure can expose information that a user should not see and create a large organizational cost. These controls cannot be treated as a separate feature added only to the user interface. They have to be reflected in ingestion, retrieval, and response generation.

### A stack built from many vendors creates support and diagnosis problems
[09:07](https://www.youtube.com/watch?v=62U6FLUCPWs&t=547s)
Ofer calls the use of separate providers for vector storage, LLMs, embeddings, keyword search, and other functions 'vendor chaos.' Each provider brings its own contract and integration work. When something fails, the team has to determine which component caused the problem. Different vendors may point at one another, while the customer still needs one issue resolved. Coordinating support across those companies can become especially difficult at enterprise scale. A more unified stack reduces the number of interfaces a team has to manage and gives it a clearer place to seek help when the query flow breaks.

### A DIY RAG platform needs a large and changing set of skills
[09:59](https://www.youtube.com/watch?v=62U6FLUCPWs&t=599s)
Ofer says a self-built RAG stack needs specialists in LLMs, embedding models, hybrid search, data engineering, metadata management, document parsing, DevOps, site reliability, and security. The team must keep ingestion and ETL flows working, operate the service, and maintain it as the underlying technology changes. These skills can be hard to find and retain as a separate group supporting several generative AI applications. He also warns that English support does not guarantee support for other languages. Every component must handle the required languages, or quality may fall when the system expands beyond English.

### A managed platform moves maintenance behind indexing and query APIs
[11:38](https://www.youtube.com/watch?v=62U6FLUCPWs&t=698s)
Vectara's approach is to provide the extraction, encoding, indexing, vector database, and retrieval components as a prebuilt platform. Customers use APIs to upload or index data and to run queries, chat, or agentic RAG applications. Ofer says the platform handles upgrades and management of the internal components. He presents accuracy, security, and observability as reasons customers choose it, with attention to retrieval quality, hallucination reduction, access controls, prompt attack prevention, and citations. The platform can run as a hosted service, in a customer's VPC, or on premises, which addresses organizations with stricter deployment requirements.

## Notable quotes
- "It turns out that building a RAG platform that is enterprise scale is much harder than most people seem to realize." (00:38)
- "Your system depends on the weakest link." (07:04)
- "The thing that frightens me the most with these kind of things when I think about real enterprise scale is how do you diagnose issues." (09:43)
- "You have to think about it upfront if you want to go to production scale as well." (11:20)

## Tools & references mentioned
- Vectara
- Notion
- Salesforce
- Hugging Face
- GPT
- LLMs
- HHM (Hallucination Evaluation Model)

## Who should watch
- You have a RAG prototype and need to estimate the work required before putting it into production.
- Your team is choosing between assembling separate retrieval, model, storage, and parsing services or adopting a managed platform.
- You need RAG answers to respect internal permissions, expose citations, support multiple languages, or run inside a VPC or on premises.

## Related talks

- [The RAG Stack We Landed On After 37 Fails](https://aietalks.com/talks/the-rag-stack-we-landed-on-after-37-fails) (Jonathan Fernandes, 18:52)
- [Scaling Enterprise-Grade RAG: Lessons from Legal Frontier](https://aietalks.com/talks/scaling-enterprise-grade-rag-lessons-from-legal-frontier) (Calvin Qi, Harvey & Chang She, LanceDB, 16:40)
- [OpenRAG: An open-source stack for RAG](https://aietalks.com/talks/openrag-an-open-source-stack-for-rag) (Phil Nash, IBM, 15:52)
- [RAG Evaluation Is Broken! Here's Why (And How to Fix It)](https://aietalks.com/talks/rag-evaluation-is-broken-heres-why-and-how-to-fix-it) (Yuval Belfer & Niv Granot, A21 Labs, 10:58)
- [RAG in 2025: State of the Art and the Road Forward](https://aietalks.com/talks/rag-in-2025-state-of-the-art-and-the-road-forward) (Tengyu Ma, MongoDB, 18:48)