# The Production AI Playbook: Deploying Agents at Enterprise Scale Sandipan Bhaumik, Databricks | AI Engineer Europe 2026 | 37:06 Source: https://www.youtube.com/watch?v=ObTPqBGsEbA Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/the-production-ai-playbook-deploying-agents-at-enterprise-scale Published: 2026-06-18 Tags: agents, evals, multi-agent, observability ## TL;DR - Production AI projects should define business success and build evaluation data before choosing a model. - Tracing every agent decision gives teams a way to diagnose failures, control costs, and meet regulatory needs. - A production incident playbook connects evaluation, tracing, containment, fixes, and a continually growing test case library. ## Summary Sandipan Bhaumik presents a five-pillar framework for moving AI agents from demos into production: evaluation, observability, data foundation, orchestration, and governance. His central advice is to define measurable business outcomes and collect real examples before debating models. Evaluation should cover deterministic checks, semantic judgments, and agent behavior such as duplicate tool calls. Tracing then makes every classification, retrieval, API call, reasoning step, and guardrail visible. Bhaumik spends particular attention on data quality and tracing-data strategy, since agents can confidently return incorrect answers. He also describes orchestrator-worker, choreography, and human-in-the-loop patterns for multi-agent systems. A retail banking chatbot case shows the approach in practice. The team selected a model in week seven of an eight-week engagement, after building evaluation and data systems. After launch, tracing exposed stale policy embeddings when customer satisfaction fell. The talk ends with an incident playbook and practical guidance for governing test cases, prompts, and evaluation costs. ## Key ideas ### Choosing a model should come after defining how success will be measured [05:01](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=301s) Bhaumik says teams often begin with a debate over GPT or Claude, then build a demo against predictable data and discover later that the system cannot handle production questions. His first pillar is evaluation. Before writing code or discussing models, teams should define success numerically for the business use case. That can include accuracy, acceptable false positives, deflection, latency, or another operational target. For a banking chatbot, the team might measure how many simple customer queries the agent handles without a human. The evaluation system should keep measuring those targets as the system changes. ### Evaluation needs deterministic, semantic, and behavioral checks [09:31](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=571s) Bhaumik describes three evaluation layers. Deterministic checks cover formats and other inexpensive rules, while classic machine learning can handle tasks such as intent classification and PII detection. Semantic checks use an LLM judge to assess safety, groundedness, and relevance against expected answers. Behavioral checks inspect what the agent did, including its tool calls and retries. An agent can return the right account balance while making three database calls instead of one. That duplication may be acceptable in a demo, but it becomes expensive when thousands of users generate queries. Teams often miss this behavioral layer. ### Tracing turns an agent response into a diagnosable sequence of actions [12:30](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=750s) In Bhaumik's retail banking example, a customer asks for an overdraft fee to be waived. The trace records intent classification, its confidence and duration, the account lookup, policy retrieval from a vector database, reasoning, guardrail checks, and the final response. Without those records, a team cannot explain what happened when the customer disputes an answer. Tracing also exposes duplicate or failing API calls. Online monitoring can trigger a fallback, limit retries to three attempts, report the issue, or pass the case to a human. Bhaumik says regulated organizations may be unable to put AI into production without this observability. ### Agents require a data strategy for both answer data and tracking data [15:15](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=915s) Bhaumik says data foundation is where he spends 60% of his project time. Human users often notice and correct bad data, while agents can confidently retrieve an incorrect value and repeat it to customers. He separates question data, which the agent uses to answer requests, from tracking data, which records traces and decisions. The second category needs its own schema and serving plan for auditors, regulators, online monitoring, and LLM judges. He describes Databricks components including Delta Lake, Unity Catalog, MLflow, Mosaic AI, and Genie as parts of a shared data foundation. A central tracing layer can collect data across frameworks and cloud platforms. ### Multi-agent systems need an explicit coordination pattern [20:13](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1213s) Bhaumik describes three orchestration patterns. In orchestrator-worker systems, a central agent assigns work to specialized agents and provides one place to inspect logs. In choreography systems, independent agents listen to a message bus and act on relevant events, allowing parallel work and lower latency. Human-in-the-loop workflows call a person when an agent falls below a confidence threshold. Moving from one agent to several increases coordination complexity, so teams also need to plan for state management and fault tolerance. Bhaumik refers viewers to a separate session covering saga, compensation, and circuit breaker patterns. ### Governance covers audit trails, privacy checks, prompts, and model changes [22:22](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1342s) Bhaumik's governance pillar asks whether the system records every action, user connection, and request. It includes pre-validation for personal information, with PII detection among the inexpensive checks. In one customer project, the team caught 47 PII breaches during testing. Prompt changes should follow enterprise change management rather than being treated as casual Git edits. Commit records should explain why a prompt changed, which failure caused the change, and what the new version should correct. Model upgrades also need testing against the organization's own evaluation data, since provider benchmarks may not predict performance on enterprise data. ### A living evaluation set turns production failures into regression tests [26:03](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1563s) For the retail banking chatbot, the team collected 200 examples of real answers from human support agents, defined an initial 60% deflection target and an approximately 85% accuracy target, and built an automated comparison pipeline. Responses below a threshold went to human review. When a problem was fixed through a prompt or tool-calling change, the case was added to the evaluation set. Bhaumik calls this a living system. It grows as the agent encounters new failures, and its rows should have owners and categories so teams can identify which kind of problem changed. ### The incident playbook links detection to containment and permanent fixes [31:06](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1866s) Bhaumik describes a production sequence: detect an issue in the evaluation dashboard, diagnose it through traces, contain it by changing or removing a prompt or routing work to a human, then fix it using the test case library and judge reports. After a bank changed its interest-rate policy, customer satisfaction fell because the chatbot still used an outdated policy document. Tracing showed that the new document had not reached the vector database because its embeddings were missing. The team corrected the data, then added the failure to the evaluation system. Bhaumik recommends connecting this process to an ITSM system so the right person receives the alert. ### Evaluation cost needs different checks in pull requests and main-branch merges [35:12](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=2112s) Behavioral evaluations can become expensive as the test set grows. Bhaumik suggests running a small subset of cases after a prompt change in continuous integration, then running the full evaluation when the change merges to the main branch. This keeps frequent checks affordable while preserving a complete test before release. The same governance applies to the evaluation library itself. Teams need an owner, categories for different failures, and records that explain how each test case relates to a security, login, tool-use, or other problem. ## Notable quotes - "Before touching any code, before discussing about any models, any features, you have to think about when we build this system, how do we measure?" (05:18) - "Agents don't forgive you. Agents will go, find it wrong, they'll give you the wrong answer confidently." (15:34) - "Your evaluation data set is a living system." (27:22) - "Once we had that system that can make AI visible, measurable, and accountable, that's when we started launching it to production." (29:00) ## Tools & references mentioned - Databricks - Amazon Web Services - GPT - Claude - MLflow - Apache Spark - Delta Lake - Unity Catalog - Delta Sharing - Mosaic AI - Genie - Agent Bricks - CrewAI - LangChain - saga pattern - compensation pattern - circuit breaker pattern ## Who should watch - You are responsible for an AI proof of concept that must survive real customer traffic and need a framework for deciding what to build before choosing a model. - Your team operates in banking, financial services, or another regulated setting and needs traceable agent decisions, PII checks, and ownership for failures. - You are building a multi-agent system and need practical guidance on evaluation data, orchestration, incident response, and the cost of repeated testing.