Generative AI applications need data that matches their specific use case, including customer information, company policies, permissions, and brand content.
2
Amazon Bedrock Knowledge Bases, Data Automation, and Guardrails provide managed pieces for processing data, building RAG applications, and controlling generated responses.
3
A production application needs evaluation, observability, optimization, semantic caching, and an automated test suite before and after changes.
Summary
Mani Khanuja argues that proprietary data determines how useful and safe an enterprise AI application can be. Different applications need different data: a travel agent needs customer profiles and travel policies, while an employee chatbot needs company information with carefully limited access. She walks through an Amazon Bedrock approach using Data Automation for multimodal data pipelines, Knowledge Bases for chunking, embeddings, ingestion, retrieval, hybrid search, reranking, and query decomposition, and Guardrails for filtering sensitive information and unwanted content. The talk then moves from construction to operation. Teams need to evaluate retrieval quality before sending context to a model, log user queries and responses, control cost and latency with semantic caching, and keep observability in place from the pilot onward. Data and retrieval strategies will change, so updates must be followed by testing against an automated suite. Her coconut metaphor captures the full loop from data preparation to production monitoring.
Generative AI changes the data work required by an application
Khanuja says generative AI has different data requirements from earlier machine learning applications. Data still needs to be transformed, parsed, and loaded, but teams also need to examine how it interacts with the technology and with users. Data silos affect applications because models can use large amounts of information from different places. The data foundation must match the business and the application. A travel agent, an employee productivity chatbot, and a marketing assistant will each need different sources, permissions, and content. The application determines what data is useful and what must remain protected.
A travel agent needs personal context and company rules
For a travel agent, personalized answers depend on a customer profile, but that profile may contain personally identifiable information that must not be disclosed. The system also needs company data, such as travel policies that determine whether a ticket refund is allowed. Airline information and related business rules add more context. Khanuja uses this example to show why personalization creates responsibility. An employee chatbot has a different requirement: it needs company information while giving an employee no more access than the request requires. The integration could be Slack or a custom application, and the source data may live in several systems.
Bedrock provides managed components for tailored applications
Khanuja presents Amazon Bedrock as a collection of building blocks beyond model choice. Data Automation can create custom data pipelines and transform content. Model customization supports fine-tuning, while model evaluation helps assess models. Knowledge Bases can speed up RAG development, and Guardrails add controls for responsible AI. She applies these capabilities to agents, RAG systems, summarization, classification, fraud detection, and contextual chatbots. Her point is that every part of the application depends on data, from the system prompt and user query to retrieved context and any data used to train or customize a model.
Knowledge Bases reduce custom RAG implementation work
Data Automation can process videos, text, and images through a single API, including financial documents with charts and line graphs. Knowledge Bases then provide native support for Data Automation, configurable chunking, embeddings, vector stores, and incremental data ingestion. Teams can use hierarchical or semantic chunking for complex content, or supply custom chunking logic. The retrieve API supports semantic and hybrid search. The retrieve-and-generate API can also apply options such as reranking and query decomposition through parameters. This lets teams adjust retrieval behavior without implementing every mechanism themselves.
A useful RAG system needs evaluation and observability
Khanuja says teams cannot optimize an application until they know what is wrong with it. They need to inspect retrieval results and evaluate context relevance before passing context to a model. Bad search results create poor responses and still incur model cost. Observability means logging user queries, retrieval hits, and model responses so the team can troubleshoot complaints and improve the system. She recommends having observability before production, including during a pilot. Evaluation metrics depend on the application. A RAG system may measure context relevance, while a summarization system needs metrics suited to summaries.
Generative AI requests are not always phrased identically, and responses are not always deterministic in their exact wording. Khanuja therefore recommends semantic caching rather than looking only for an exact repeat of a question. When a similar question has already been answered, the application can retrieve the cached result instead of invoking the foundation model again. That can reduce charges and avoid the latency involved in having the model synthesize another response. Cost and latency belong alongside accuracy when teams optimize application performance.
Production changes require fresh data and automated tests
Evaluation may show that the application needs newer data or a different retrieval strategy. After making an update, the team must test again rather than assuming the change worked. Khanuja recommends a defined test suite so evaluations can run automatically whenever the data, chunking, or other application behavior changes. Repeated testing lowers the chance of introducing a problem into production and raises the chance that the deployed application maintains its quality. The work continues after the first release because data becomes stale and application strategies need revision.
"Personalized experience and personalized experience require personalized data and it also brings up lot of responsibility because you can no longer afford to have PII information disclosed."03:39
Who should watch
You are building an enterprise RAG or chatbot application and need to connect proprietary data to model responses without exposing customer or employee information.
Your team has a prototype but lacks retrieval evaluation, request and response logging, or a repeatable way to test changes before deployment.
You are deciding whether managed Bedrock features can replace custom work for multimodal data processing, chunking, retrieval, and response controls.
