# From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work Sandipan Bhaumik, Databricks | AI Engineer Europe 2026 | 26:29 Source: https://www.youtube.com/watch?v=2czYyrTzILg Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/from-chaos-to-choreography-multi-agent-orchestration-patterns-that-actually-work Published: 2026-04-08 Tags: agents, multi-agent, observability, reliability, workflows ## TL;DR - Adding agents turns an AI feature into a distributed system with race conditions, stale reads, and cascading failures. - Choreography fits independent event-driven workflows, while orchestration fits complex dependencies that need central control, retries, state tracking, and rollback. - Immutable versioned state, data contracts, circuit breakers, and compensation logic give multi-agent workflows production safeguards. ## Summary Sandipan Bhaumik argues that multi-agent systems fail for distributed-systems reasons rather than because of weak models or prompts. He describes a credit decisioning deployment where stale cached credit scores caused 20% of decisions to receive incorrect risk ratings. The talk compares choreography, where agents communicate through events, with orchestration, where a central workflow controls calls, state, retries, and parallel work. Bhaumik recommends immutable, versioned state snapshots instead of shared mutable records, with schemas checked at every handoff. He also covers circuit breakers for repeated agent failures and saga-style compensation for undoing completed work after a later step fails. His Databricks architecture uses LangGraph, Unity Catalog, Delta Lake, MLflow, and serving-layer controls. The practical message is direct: scaling from one agent requires systems engineering, observability, and explicit failure handling. ## Key ideas ### Adding agents creates distributed-systems problems [00:00](https://www.youtube.com/watch?v=2czYyrTzILg&t=0s) Bhaumik says teams often treat extra agents like extra product features, then discover coordination failures. In his credit decisioning example, a credit score agent wrote 750 to PostgreSQL, while the risk agent read 680 from a cache 500 milliseconds later. The database write succeeded, but cache invalidation failed. After three days in production, 20% of decisions had incorrect risk ratings, and the team needed two days to find the cause. The model and prompts were not the problem. Shared caches, concurrent agents, and missing coordination created the race condition. ### Coordination complexity grows through agent relationships [04:00](https://www.youtube.com/watch?v=2czYyrTzILg&t=240s) Moving from one agent to five agents creates more than five times the coordination work, according to Bhaumik. One agent has no agent-to-agent coordination. Two agents have at least one connection, while five agents have at least 10 possible connections. Each connection can introduce a failure point, a race condition, or a state synchronization problem. The system also gains more possible failure modes as agents depend on one another. The architecture therefore needs explicit coordination patterns before the team adds more agents. ### Choreography works for autonomous event-driven workflows [05:58](https://www.youtube.com/watch?v=2czYyrTzILg&t=358s) In a choreographed workflow, agents publish and consume events through a message bus. A research agent publishes a research-completed event, an analysis agent consumes it and publishes analysis-ready, and a report agent consumes that result. No central coordinator directs the sequence. This keeps agents loosely coupled and makes it easier to add agents that subscribe to existing events. The cost is debugging. Operators must determine which agent failed to publish, whether an event was consumed, and whether it was consumed twice. Bhaumik says choreography requires strong event tracing and delivery guarantees. ### Orchestration gives complex workflows one execution authority [08:28](https://www.youtube.com/watch?v=2czYyrTzILg&t=508s) With orchestration, a central workflow component calls agents directly, waits for results, runs independent agents in parallel, and combines their outputs for later steps. Agents do not call one another. The orchestrator owns the execution graph, state, retries, logs, and workflow status. Bhaumik recommends this pattern when dependencies are complex, rollback matters, a workflow needs one system-wide dashboard, or the workflow is stable. He says financial-services teams favor orchestration because a failed credit decision must be traceable to a specific agent, sequence, and data input. ### Immutable versioned state prevents shared-record races [11:20](https://www.youtube.com/watch?v=2czYyrTzILg&t=680s) Bhaumik rejects shared mutable state as the default for multi-agent workflows. Two agents can read the same value and write different results, allowing the last write to erase the other update. His alternative is an append-only sequence of immutable state snapshots. Agent A creates version one, Agent B validates it and creates version two, and each later agent appends another version instead of modifying an earlier record. A failed agent can roll back to the previous version, while operators can inspect the entire state evolution. Version numbers and creator information make it possible to trace a bad output back through earlier inputs. ### Data contracts catch bad handoffs at the boundary [15:23](https://www.youtube.com/watch?v=2czYyrTzILg&t=923s) Each agent should declare what it outputs and what the next agent accepts. Bhaumik's example has a research agent producing findings, a confidence score, sources, and a timestamp. The analysis agent validates that structure and rejects the handoff when confidence is below 0.7. This exposes low-quality data immediately, instead of allowing it to travel through several agents and produce a bad report. In Databricks deployments, he describes registering input and output schemas in Unity Catalog so contracts are versioned and governed centrally. ### Failure handling needs circuit breakers and compensation [16:30](https://www.youtube.com/watch?v=2czYyrTzILg&t=990s) Agents will time out, hit rate limits, or crash during a workflow, so Bhaumik recommends designing for those cases. A circuit breaker opens after repeated failures, such as five failures in a row, and fails fast instead of sending more requests to a broken agent. After a timeout such as 60 seconds, it tests one request in a half-open state. For partial workflows, the saga or compensation pattern gives each agent an execute method and a compensating method. If a later step fails, the orchestrator walks backward through successful steps and reverses their work. ### A production architecture centralizes state, tracing, and controls [21:01](https://www.youtube.com/watch?v=2czYyrTzILg&t=1261s) Bhaumik's Databricks architecture puts LangGraph and the Mosaic AI Agent Framework in the orchestration layer. Agents can be Unity Catalog functions or registered models, exposed through Model Serving or Function Serving. Delta Lake stores customer data and immutable state versions, while MLflow Traces ties agent runs to those versions and records inputs, outputs, latency, and token usage. Unity Catalog provides access control, lineage, and audit information. Circuit breakers, retries, timeouts, and rate limits sit at the serving layer, and failed steps can trigger compensation through the orchestrator. ## Notable quotes - "It's building a distributed system." (00:19) - "The race condition wasn't in the database, it was in the architecture." (03:28) - "If you can't trace events through your system, choreography will destroy you." (07:45) - "Every operation must be reversible." (20:23) - "In production, you have to build systems, and systems are hard." (25:25) ## Tools & references mentioned - AWS - Databricks - LangGraph - Mosaic AI Agent Framework - Agent Bricks - Unity Catalog - Delta Lake - MLflow - PostgreSQL ## Who should watch - You have moved from one agent to several and are seeing stale data, silent handoff failures, or workflows that are hard to explain. - You are choosing between event-driven choreography and a central orchestrator for a production agent system. - You need state versioning, schema checks, retries, circuit breakers, or rollback behavior around agent calls. ## Editor's note Sandipan Bhaumik describes a credit decisioning race in which a stale cached score led to incorrect risk ratings in production, and the team needed two days to find the cause. Kitaru records every model call and tool result from a real run, then replays the agent against those same inputs and responses after a change to the code, prompt, or model. Written by the AIE Talks editors (the Kitaru team), not by the speaker. ## Related talks - [AgentCraft: Putting the Orc in Orchestration](https://aietalks.com/talks/agentcraft-putting-the-orc-in-orchestration) (Ido Salomon, 11:18) - [The Multi-Agent Architecture That Actually Ships](https://aietalks.com/talks/the-multi-agent-architecture-that-actually-ships) (Luke Alvoeiro, Factory, 18:31) - [Scaling AI Agents Without Breaking Reliability](https://aietalks.com/talks/scaling-ai-agents-without-breaking-reliability) (Preeti Somal, Temporal, 15:01) - [Agents vs Workflows: Why Not Both?](https://aietalks.com/talks/agents-vs-workflows-why-not-both) (Sam Bhagwat, Mastra, 15:37) - [Conquering Agent Chaos](https://aietalks.com/talks/conquering-agent-chaos) (Rick Blalock, Agentuity, 14:40)