An enterprise RAG system requires much more than a vector database and an LLM, including parsing, chunking, retrieval, evaluation, access control, and monitoring.
2
DIY RAG stacks can develop latency, scaling, cost, security, vendor coordination, staffing, and language-support problems as they move into production.
3
A turnkey RAG platform hides and maintains these components while providing APIs for indexing and querying, with deployment options in a customer's VPC or on premises.
Summary
Ofer explains why a RAG proof of concept can become difficult to operate at enterprise scale. A production system must ingest varied sources, parse documents and tables, chunk text, create embeddings, search with vectors and keywords, rerank results, generate answers, detect hallucinations, and preserve citations. Each part adds work and can affect latency, cost, quality, security, and support. He describes seven problem areas: response quality, latency, scaling and cost, security and compliance, vendor coordination, staffing and maintenance, and non-English language support. Vectara's alternative is a managed RAG platform with APIs for indexing and querying, plus deployment in a customer's VPC or on premises. Ofer also describes Vectara's hallucination evaluation model, access controls, prompt attack prevention, and observability. His argument is practical: teams should account for the ongoing engineering and operational work behind a RAG stack, rather than judging the effort by the first working demo.
A RAG system has separate ingestion and query flows
Ofer defines RAG as a way to ground an LLM in a company's own data. During ingestion, the system extracts content from documents or databases, chunks text into smaller pieces, creates embeddings, and stores those embeddings in a vector database. During a query, the question is embedded and matched against the stored vectors. Production systems often add hybrid search, which combines vector and keyword search, plus reranking based on relevance, diversity, or business rules. The retrieved chunks are sent to the LLM with a prompt. The system may then check for hallucinations and preserve the source path so the answer can include citations.
Response quality requires ongoing work after the first demo
Building a high-quality RAG system requires more than selecting an embedding model and connecting a vector store. Ofer says teams must invest in parsing, chunking strategies, table parsing and table understanding, hybrid search, and other retrieval components. They also need continuous evaluation as data is added or refreshed and as system components change. The target is for answer quality to remain the same or improve over time. Hallucination prevention and correction add another layer of complexity. Explainability also needs engineering support, because the application must retain where retrieved facts came from so users can inspect the source.
Multiple retrieval components can make latency difficult to control
Ofer says latency often rises because a query passes through several components that are not fully coordinated. A simple vector search may be fast, while hybrid search and additional rankers add more processing. The total flow is constrained by its slowest component. If one service has a latency problem, the complete query experiences that delay. This makes production latency harder to predict than the initial prototype suggests. Users expect answers quickly, so the system needs more than a fast vector database. The whole retrieval and generation path has to work within the required response time.
Production scale raises both infrastructure and token costs
A production RAG stack can include embedding models, LLMs, vector databases, document-parsing services, table-understanding tools, and other supporting components. Ofer says maintaining low latency and good quality requires capable versions of these components, which brings costs for GPUs, CPUs, storage, and external services. The cost profile changes as a system grows from a small collection of documents to thousands, hundreds of thousands, or millions. Token costs also become harder to maintain. The concern is therefore the combined operating cost of the full stack, rather than the price of any single model or database.
Access controls and sensitive data must follow the RAG flow
Ofer focuses on attribute-based access control as a common enterprise requirement. If documents are visible only to a particular department or executive, the RAG system must apply the same restrictions when it retrieves and presents information. Sensitive data such as PHI and PII must also be ingested and handled correctly throughout the stack. A failure can expose information that a user should not see and create a large organizational cost. These controls cannot be treated as a separate feature added only to the user interface. They have to be reflected in ingestion, retrieval, and response generation.
A stack built from many vendors creates support and diagnosis problems
Ofer calls the use of separate providers for vector storage, LLMs, embeddings, keyword search, and other functions 'vendor chaos.' Each provider brings its own contract and integration work. When something fails, the team has to determine which component caused the problem. Different vendors may point at one another, while the customer still needs one issue resolved. Coordinating support across those companies can become especially difficult at enterprise scale. A more unified stack reduces the number of interfaces a team has to manage and gives it a clearer place to seek help when the query flow breaks.
A DIY RAG platform needs a large and changing set of skills
Ofer says a self-built RAG stack needs specialists in LLMs, embedding models, hybrid search, data engineering, metadata management, document parsing, DevOps, site reliability, and security. The team must keep ingestion and ETL flows working, operate the service, and maintain it as the underlying technology changes. These skills can be hard to find and retain as a separate group supporting several generative AI applications. He also warns that English support does not guarantee support for other languages. Every component must handle the required languages, or quality may fall when the system expands beyond English.
A managed platform moves maintenance behind indexing and query APIs
Vectara's approach is to provide the extraction, encoding, indexing, vector database, and retrieval components as a prebuilt platform. Customers use APIs to upload or index data and to run queries, chat, or agentic RAG applications. Ofer says the platform handles upgrades and management of the internal components. He presents accuracy, security, and observability as reasons customers choose it, with attention to retrieval quality, hallucination reduction, access controls, prompt attack prevention, and citations. The platform can run as a hosted service, in a customer's VPC, or on premises, which addresses organizations with stricter deployment requirements.
"The thing that frightens me the most with these kind of things when I think about real enterprise scale is how do you diagnose issues."09:43
Who should watch
You have a RAG prototype and need to estimate the work required before putting it into production.
Your team is choosing between assembling separate retrieval, model, storage, and parsing services or adopting a managed platform.
You need RAG answers to respect internal permissions, expose citations, support multiple languages, or run inside a VPC or on premises.
