Adding agents turns an AI feature into a distributed system with race conditions, stale reads, and cascading failures.
2
Choreography fits independent event-driven workflows, while orchestration fits complex dependencies that need central control, retries, state tracking, and rollback.
3
Immutable versioned state, data contracts, circuit breakers, and compensation logic give multi-agent workflows production safeguards.
Summary
Sandipan Bhaumik argues that multi-agent systems fail for distributed-systems reasons rather than because of weak models or prompts. He describes a credit decisioning deployment where stale cached credit scores caused 20% of decisions to receive incorrect risk ratings. The talk compares choreography, where agents communicate through events, with orchestration, where a central workflow controls calls, state, retries, and parallel work. Bhaumik recommends immutable, versioned state snapshots instead of shared mutable records, with schemas checked at every handoff. He also covers circuit breakers for repeated agent failures and saga-style compensation for undoing completed work after a later step fails. His Databricks architecture uses LangGraph, Unity Catalog, Delta Lake, MLflow, and serving-layer controls. The practical message is direct: scaling from one agent requires systems engineering, observability, and explicit failure handling.
Bhaumik says teams often treat extra agents like extra product features, then discover coordination failures. In his credit decisioning example, a credit score agent wrote 750 to PostgreSQL, while the risk agent read 680 from a cache 500 milliseconds later. The database write succeeded, but cache invalidation failed. After three days in production, 20% of decisions had incorrect risk ratings, and the team needed two days to find the cause. The model and prompts were not the problem. Shared caches, concurrent agents, and missing coordination created the race condition.
Coordination complexity grows through agent relationships
Moving from one agent to five agents creates more than five times the coordination work, according to Bhaumik. One agent has no agent-to-agent coordination. Two agents have at least one connection, while five agents have at least 10 possible connections. Each connection can introduce a failure point, a race condition, or a state synchronization problem. The system also gains more possible failure modes as agents depend on one another. The architecture therefore needs explicit coordination patterns before the team adds more agents.
Choreography works for autonomous event-driven workflows
In a choreographed workflow, agents publish and consume events through a message bus. A research agent publishes a research-completed event, an analysis agent consumes it and publishes analysis-ready, and a report agent consumes that result. No central coordinator directs the sequence. This keeps agents loosely coupled and makes it easier to add agents that subscribe to existing events. The cost is debugging. Operators must determine which agent failed to publish, whether an event was consumed, and whether it was consumed twice. Bhaumik says choreography requires strong event tracing and delivery guarantees.
Orchestration gives complex workflows one execution authority
With orchestration, a central workflow component calls agents directly, waits for results, runs independent agents in parallel, and combines their outputs for later steps. Agents do not call one another. The orchestrator owns the execution graph, state, retries, logs, and workflow status. Bhaumik recommends this pattern when dependencies are complex, rollback matters, a workflow needs one system-wide dashboard, or the workflow is stable. He says financial-services teams favor orchestration because a failed credit decision must be traceable to a specific agent, sequence, and data input.
Immutable versioned state prevents shared-record races
Bhaumik rejects shared mutable state as the default for multi-agent workflows. Two agents can read the same value and write different results, allowing the last write to erase the other update. His alternative is an append-only sequence of immutable state snapshots. Agent A creates version one, Agent B validates it and creates version two, and each later agent appends another version instead of modifying an earlier record. A failed agent can roll back to the previous version, while operators can inspect the entire state evolution. Version numbers and creator information make it possible to trace a bad output back through earlier inputs.
Each agent should declare what it outputs and what the next agent accepts. Bhaumik's example has a research agent producing findings, a confidence score, sources, and a timestamp. The analysis agent validates that structure and rejects the handoff when confidence is below 0.7. This exposes low-quality data immediately, instead of allowing it to travel through several agents and produce a bad report. In Databricks deployments, he describes registering input and output schemas in Unity Catalog so contracts are versioned and governed centrally.
Failure handling needs circuit breakers and compensation
Agents will time out, hit rate limits, or crash during a workflow, so Bhaumik recommends designing for those cases. A circuit breaker opens after repeated failures, such as five failures in a row, and fails fast instead of sending more requests to a broken agent. After a timeout such as 60 seconds, it tests one request in a half-open state. For partial workflows, the saga or compensation pattern gives each agent an execute method and a compensating method. If a later step fails, the orchestrator walks backward through successful steps and reverses their work.
A production architecture centralizes state, tracing, and controls
Bhaumik's Databricks architecture puts LangGraph and the Mosaic AI Agent Framework in the orchestration layer. Agents can be Unity Catalog functions or registered models, exposed through Model Serving or Function Serving. Delta Lake stores customer data and immutable state versions, while MLflow Traces ties agent runs to those versions and records inputs, outputs, latency, and token usage. Unity Catalog provides access control, lineage, and audit information. Circuit breakers, retries, timeouts, and rate limits sit at the serving layer, and failed steps can trigger compensation through the orchestrator.
