Enterprise AI gets business value when systems can handle the company's specific context and data.
2
Production RAG requires designing for scale, security, compliance, real users, and many use cases from the beginning.
3
Teams should release early, integrate AI into existing workflows, track failures with attribution, and aim at problems where solving them creates real business value.
Summary
Douwe Kiela argues that enterprise AI is held back by a context problem. Language models can write code and solve difficult problems, but they still need the right company data and domain context to produce useful answers. RAG systems provide that context, yet the model is only one part of the product. Kiela describes lessons from deploying RAG agents at enterprise scale: focus on the whole system, specialize around company expertise, work with noisy data, and design for production from the first day. Pilots are easy compared with serving thousands of users across large document collections while meeting security and compliance requirements. He recommends releasing barely functional systems to real users, integrating them into existing workflows, and creating an early moment where users discover clear value. Accuracy alone is insufficient. Teams also need observability, evidence, and audit trails that explain why a system produced an answer. He ends by asking teams to pursue ambitious business problems rather than small internal conveniences.
Enterprise AI needs context to create differentiated value
Kiela describes a context paradox in enterprise AI. Language models can generate code and solve mathematical problems, yet they struggle to place information in the right company-specific situation. Humans do this through expertise and intuition developed over time. General assistants can improve convenience and productivity, but businesses want differentiated value and wider transformation. Kiela's point is that the further a company moves toward that value, the more carefully it must handle its internal context. He connects this observation to the work of Contextual AI, which he says was started to help bridge that gap.
The system around the model determines whether RAG solves the problem
Kiela says language models are often only 20% of a larger enterprise AI system. RAG is a standard way to make generative models work with company data, but teams often focus on a newly released model and overlook retrieval, data handling, evaluation, and the rest of the pipeline. A mediocre model with an excellent RAG pipeline can outperform an excellent model with a poor pipeline. His practical advice is to think about the system that solves the business problem instead of treating the model as the product.
Company expertise is the fuel for useful enterprise systems
Enterprises contain institutional knowledge that general-purpose assistants struggle to match. Kiela says companies should try to unlock that expertise and specialize systems around it. Contextual AI calls this idea 'specialization over AGI'. General intelligence has many uses, but a difficult, domain-specific problem is easier to solve when the system is built around a clearly understood use case and the company's own knowledge. The company data also becomes more important over time than its transient workforce. Kiela calls that data the company's moat, while warning that making noisy data usable at scale is extremely difficult.
A successful pilot says little about production readiness
Kiela calls pilots easy and production much harder. A team can use a framework, add documents, and give a working RAG system to ten friendly users. The situation changes when the system must handle tens of thousands, hundreds of thousands, or millions of documents, thousands of users, many different use cases, security, and compliance. He says existing open-source tools make this difficult at enterprise scale. Teams should design for production from day one instead of building a pilot and discovering later that the proposed rollout has much larger requirements.
Early release and real feedback beat waiting for a perfect system
Speed matters more than perfection during production rollouts of RAG agents. Kiela recommends giving an early version to real users rather than only friendly testers. The system can be barely functional at first, because direct feedback lets the team improve it step by step. He describes this as hill climbing toward a system that is good enough. Waiting until the product seems perfect makes the gap between a pilot and production harder to cross. Engineers also need to spend their time on business value instead of repeated low-level decisions such as chunking strategies and basic prompt design that platforms can abstract away.
A system can run in production and still have almost no users. Kiela says this happens when model-risk processes make the result barely useful, or when people do not know how to use the technology. The solution includes making AI easy to consume and placing it close to workflows that already exist inside the company. He gives Qualcomm as an example, where thousands of customer engineers use the system globally. One user found a seven-year-old document that answered long-standing questions. Kiela sees this kind of discovery as the moment when users understand why the system matters.
Teams must manage wrong answers with evidence and audit trails
Kiela says accuracy is close to a baseline requirement for enterprise systems. The harder operational question is what to do about the remaining errors. He recommends observability, careful evaluation, and audit trails, especially in regulated industries. A RAG system should be able to show why it generated an answer and identify the document where its evidence came from. Teams can also postprocess generated claims and check that the claims have support. Attribution therefore becomes a practical way to investigate and manage inaccuracy after deployment.
Ambitious use cases are more likely to produce real ROI
Kiela says projects often fail because teams aim too low. A system that answers basic questions about a 401k provider or vacation allowance may be useful, but it does not necessarily create meaningful business value. He asks teams to choose problems where solving them would produce real ROI rather than a gimmick that people rarely use. His final advice is to be ambitious, build around the expertise inside the company, and avoid settling for general solutions or only the easiest internal tasks.
"You can have a relatively mediocre language model but an amazing RAG pipeline around it and that's going to be much better than an amazing language model with a terrible RAG pipeline around it."05:11
Who should watch
You are responsible for moving a RAG or agent prototype into an enterprise deployment with many users and large document collections.
Your team has built an AI system that technically works, but adoption is low or risk and compliance processes have made it hard to use.
You need to choose an enterprise AI use case and want a direct argument for focusing on company expertise, measurable value, and ambitious problems.
