Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4)

Priyanka Phatak, Anthropic, Gabe Cemaj, Anthropic18:58 · Oct 2026 · 3,712 views
Thumbnail for Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) Watch on YouTube
TL;DR
  1. 1

    Agent definitions can live in YAML and be synchronized across workspaces through CI/CD with the ant CLI.

  2. 2

    Tenant isolation can use separate agents, workspaces, organizations, or session metadata, depending on the required boundary.

  3. 3

    A coordinator agent can keep receiving messages while subagents handle custom tools and continue working.

Summary

This closing Q&A covers practical limits and design choices for Claude Managed Agents. Priyanka Phatak and Gabe Cemaj describe keeping durable agent resources in YAML, then syncing them through CI/CD with the ant CLI. They explain that custom tool waits block new messages, while other work can accept queued messages. Anthropic runs large internal eval sets against new models and recommends that customers start with well-defined, high-quality data, including historical sessions. Workspace access is controlled by organization admins, though resource sharing is not fine-grained by default. For tenant isolation, customers can use separate workspaces or organizations. GPU and memory requirements call for self-hosted sandboxes through providers such as Modal. The speakers also describe mapping Slack threads to sessions, running long tasks until an exit criterion is met, and using a coordinator pattern so a subagent can wait on custom tools without making the main agent unavailable.

Key ideas
00:01

Agent definitions can be versioned in YAML and synced through CI/CD

The speakers describe the ant CLI as an API wrapper that supports Terraform-like operations with YAML. Teams can keep durable resources in their codebase and run CI/CD whenever an agent resource changes. Anthropic uses the same approach for long-term resources. This gives teams a source-controlled definition that can be synchronized to the managed service instead of edited only through an interface.

01:04

A custom tool wait blocks incoming messages

When an agent is waiting for a custom tool result, execution is paused and the caller is expected to return that result. New messages are allowed while the model is doing other work, but they cannot be queued during the custom tool wait because of ordering. Sending a message in that state returns a 400 response.

02:42

Eval quality depends on the data and problem definition

Anthropic runs internal evals as new models and capabilities become available. Its harness uses different datasets and can run many sessions in parallel, then tune prompts and assumptions for each model release. For customer-built agents, the advice is to begin with high-quality evaluation data. The speakers warn that an ill-defined task can produce a perfect score without measuring useful performance. Historical sessions can provide material for building and scoring a dataset.

04:37

Workspace access is the main organization-level sharing boundary

Agent resources are bound to a workspace, and API keys within that workspace can access them. An organization admin creates workspaces and decides who can use each one. The platform does not provide fine-grained controls to share one resource while hiding another. Builders can still expose partial capabilities through their own integrations, depending on how they design the end-user interface.

06:30

Code-review agents can run many sessions against a pull request

An integration can start one or many managed-agent sessions when a pull request opens. Subagents can inspect changes and find bugs, while a review loop can add comments and revisit the code. The speakers say this has reduced per-pull-request review time and bugs internally, although the growth in agent-generated pull requests creates an unresolved capacity problem. A sandbox can also run the application so Claude can validate its own changes as a QA agent.

08:48

Tenant isolation should use separate workspaces when data separation is required

A provider can model customers with an agent per customer, a workspace per customer, or an organization per customer. If one agent definition must be shared, it can be synchronized from YAML to multiple workspaces through automation. A single workspace can instead use a customer ID in session metadata, but the speakers recommend individual workspaces when customers must not share data. Resources are allocated per workspace.

12:10

Custom memory and GPU needs require self-hosted sandboxes

The managed service's provided sandboxes have the same environment. For higher memory, GPU access, or other resource allocations, the speakers recommend self-hosted sandboxes. They mention Modal and other providers, with cookbooks and integrations that run the agent loop in the external environment. The provider documentation contains the available environment specifications and integration guidance.

16:30

A coordinator can keep the main session available while a subagent waits

For an agent that must stay responsive while a custom tool is waiting, the speakers suggest a coordinator pattern. The main agent delegates the custom tool work to a subagent. The subagent emits the tool call and receives the result on its own thread, while the user interacts with the main thread. The threads belong to the same core session, and the platform routes results using IDs. The main thread can be idle while subagents continue working or wait for tools.

"So, you can put your long-term resources in YAML in your code base and then build CI/CD to basically update the ant with ant CLI using those YAMLs every time you make a change to your resource."Priyanka Phatak00:01
Who should watch
  • You are putting agent definitions into production and need a versioned CI/CD workflow for updating them.
  • Your application serves several customers and you need to choose between session metadata, workspaces, and separate organizations.
  • You are building long-running or interactive agents that need custom tools, GPUs, or a responsive coordinator.