A retail copilot can combine product retrieval from Azure AI Search with customer and order data from Cosmos DB before sending the context to an OpenAI model.
2
Prompt flow lets developers build and test the RAG workflow locally, then deploy the same flow as a hosted endpoint through Azure AI Studio.
3
LLM applications need evaluation sets that measure relevance, groundedness, coherence, and fluency, followed by prompt, retrieval, and data changes when scores are poor.
Summary
David Smith and Cedric Vidal lead a hands-on workshop for building a retail chatbot with Azure AI. The application answers questions about products and a logged-in customer's order history. Azure AI Search retrieves relevant product documentation through vector search, Cosmos DB supplies customer data, and an OpenAI model generates the response from the assembled prompt. The workshop uses GitHub Codespaces for development and Prompt flow for connecting inputs, retrieval, database lookup, prompt construction, and model execution. The flow runs locally during development and is then deployed as a cloud endpoint. A large part of the talk covers evaluation. GPT-4 judges outputs from a cheaper GPT-3.5 Turbo application for relevance, groundedness, coherence, and fluency. The presenters show that the system can invent a product when asked about something absent from the catalogue. They recommend testing expected and adversarial questions, then iterating on prompts, chunking, retrieval, and monitoring before production.
The workshop builds a retail copilot around two kinds of business data
The application is a chatbot for a camping and outdoor retailer. It can answer product questions using information such as features, FAQs, return policies, technical specifications, and care instructions. It can also answer questions about a logged-in customer's purchases. In the demonstration, Sarah Lee asks for a tent that pairs with Trail Walker shoes, then asks what she has already purchased. The first answer comes from product information, while the second uses her order history. The workshop focuses on the backend, with frontend code and data available in the GitHub repository.
The RAG flow combines vector retrieval with a customer database lookup
A customer question and chat history go to the endpoint. The question is embedded into a multidimensional vector, then compared with embedded product information in Azure AI Search. The most relevant product documents are retrieved and placed into the language model's context. At the same time, customer information and purchase history come from Cosmos DB. Prompt flow combines the question, product context, customer details, and order history into a prompt before the OpenAI model generates the answer. The result is returned to the website as chatbot text.
Prompt flow is designed to connect resources into a deployable endpoint
David describes Prompt flow as a development tool inside Azure AI Studio. In this example, it orchestrates embedding, retrieval from AI Search, a Cosmos DB lookup, prompt construction, and response generation. The flow is represented as a directed acyclic graph, so the graph itself cannot contain loops or conditionally skip nodes. Individual Python nodes can contain normal Python conditionals and loops. Prompt flow can be written through YAML, a visual representation, or a more programmatic Python approach. It can also run as a library, although this workshop deploys it as an endpoint.
The local development loop keeps orchestration fast while cloud resources remain available
The workshop first runs the flow inside GitHub Codespaces. The orchestration runs in the Codespaces environment, while the database and OpenAI endpoint remain in Azure. This gives developers a quick place to try prompts, check connections, and replace model steps. After the local work, the same flow is published into Azure, where it runs in a containerized application for production use. Cedric explains that deployment can handle the environment and scaling, while developers add packages when their flow needs extra processing.
Azure AI Studio and Copilot Studio have different jobs
David distinguishes Azure AI Studio from Copilot Studio. AI Studio is used to create and evaluate model endpoints, connect them to services such as Cosmos DB and AI Search, and manage the resources around a flow. Copilot Studio is aimed at building complete applications and user interfaces, including applications integrated with Teams. A Copilot Studio application can call an endpoint created in AI Studio when more customization is needed. Prompt flow can also be exported from the AI Studio playground after configuring a RAG application.
Vector search at scale often works better when combined with keyword search
When asked about searching a million vectors, David explains that vector databases are indexed to support fast nearest-neighbor searches. He also gives a practical qualification from the team's experience: vector search alone has often been insufficient for real applications. A hybrid combination of keyword and vector search has performed better than either method alone. Azure AI Search supports this combination. The workshop keeps the example simple with vector search, and the product markdown files are indexed without chunking in this particular setup. David says Azure AI Search can chunk data in other workflows.
The example's database access is concentrated in the flow rather than the web app
The web application sends the customer's user ID and question to the Prompt flow endpoint. In the workshop's implementation, the flow uses a privileged connection to query Cosmos DB with that ID. The application itself does not directly search the database. The retrieved orders are inserted into the model prompt, and only the generated answer goes back to the web app. The presenters discuss delegated identity and row-level access as an area that requires a different authentication design if the flow must inherit the calling user's permissions.
Evaluation uses a stronger model to judge a cheaper production model
The workshop evaluates a GPT-3.5 Turbo RAG application with GPT-4. The evaluator receives the question, retrieved context, and generated answer, then scores qualities such as relevance, groundedness, coherence, and fluency. The presenters demonstrate a failure with a question about a bright toothbrush. The retailer does not sell toothbrushes, but GPT-3.5 Turbo invents a product called a Fresh Breeze travel toothbrush. GPT-4 gives the answer a groundedness score of one out of five and a relevance score of one, while its English remains coherent and fluent.
LLM Ops is an iterative process from evaluation through monitoring
David describes an LLM Ops cycle that begins with choosing a use case and exploring the available data. Developers build an initial flow, run it against sample questions, and evaluate the results. If scores are poor, they can change prompts, alter the RAG process, or chunk and present data differently. Later testing can include a user cohort, testers, and a red team. Once deployed, the system should monitor samples of real questions and answers, watching for changes in groundedness and other scores. Product changes and adversarial users can trigger another development and evaluation cycle.
"The advantage of doing it within a vector database is you can do that quickly in that scale and at speed."1:29:25
Who should watch
You are building a RAG chatbot that needs to combine retrieved documents with customer-specific records.
Your team needs a code-first workflow for testing prompts and connections locally before deploying an endpoint in Azure.
You want a concrete example of evaluating hallucinations and tracking RAG quality rather than relying on ordinary pass-or-fail software tests.
