Semi-autonomous multi-agent systems need interfaces that expose capabilities, activity, interruptions, and action costs to users.
2
Teams should define the task, build a non-agent baseline, create tools, and set up evaluations before building the agent.
3
Multi-agent designs fit tasks that need planning, multiple perspectives, extensive context processing, or adaptation as the environment changes.
Summary
Victor Dibia presents BlenderLM, a multi-agent system he built from scratch to turn natural-language requests into Blender scenes. The demo shows a planner, an execution loop, streamed activity, and a verifier that checks scene state and progress. From this example, he extracts four interface principles: help users discover what the system can do reliably, expose activity and provenance, allow users to pause or roll back work, and estimate the cost or risk of actions before delegating them. He also argues for an eval-driven development process. Start with a clear task, a non-agent baseline, suitable tools, and a test bed. Add planning and verification only when evaluation shows that they improve the result. Dibia is careful about the limits of the approach. Multi-agent systems increase autonomy and also increase the surface for error, so teams should use them only when the task benefits from adaptive, exploratory work.
Exploratory tasks need a different structure from fixed workflows
Dibia contrasts deterministic workflows with autonomous exploratory systems. A fixed workflow works well when the developer knows the exact sequence of steps and can use function calling or structured output to make the system reliable. Blender tasks are different because each action changes the environment. Clicking something can alter the whole scene, so the next step depends on what happened. In the exploratory design, an LLM drives the flow of control, calls tools, observes their results, and decides how to continue. The system needs autonomy, tools that can have side effects, and the ability to break complex tasks into steps that run for an extended period.
The BlenderLM interface makes agent work visible while it happens
BlenderLM connects a web application to a real Blender instance through a WebSocket connection. The interface includes fixed tools that a developer can trigger directly, such as clearing the scene. For an instruction to create two glossy silver balls, the UI streams activity as the system analyzes the request, creates a plan, calls tools, and updates Blender. The planner describes steps such as adding a ground plane, creating separated spheres, and assigning a material. A verifier then takes a snapshot, reads the scene contents, and uses a language model to judge whether the task is progressing or stalled. Users can inspect the resulting scene directly in Blender.
A good agent starts with the task and baseline, not the agent definition
Dibia recommends a build order for agent systems. First define the goal. Then create a baseline that does not depend on agents or AI, so the basic application works correctly. Next identify the tools a person would need to complete the task. After that, define a test bed for evaluation. Only then should the team build the agent. For BlenderLM, the baseline was a script that created a single cube, supported by a Blender add-on and a client library for socket connections. This order gives the team something concrete to test before adding an LLM loop, planning, or verification.
BlenderLM uses task-specific tools, such as one that creates a Blender object, alongside a general-purpose tool that executes arbitrary code. The general tool lets the language model generate Python that drives Blender, while the specific tools constrain common operations. Dibia says the agent is only as good as the tools it receives, so tool design deserves about half of the development time. He suggests testing the tools in code before relying on them inside the agent. The point is practical: adding more autonomy cannot compensate for tools that are incomplete, unsafe, or difficult to evaluate.
Planning and verification are separate additions to the basic agent loop
The first version of the agent is a language model in a loop with function calls. Dibia treats that as a starting point rather than a complete system. The verifier runs after each step, collecting the current scene contents, the list of objects, and visual information. It predicts whether the system is making progress or has completed the task, which helps determine what should happen next. The planner breaks the user request into atomic steps, such as setting up the environment before creating and styling objects. Dibia added these agents iteratively and used an interactive evaluation tool to check whether each addition improved the application.
Dibia's first interface principle is capability discovery. An agent may be able to attempt many actions, but only some are reliable enough to promise to users. The interface should itemize the tasks the agent can perform with high reliability. In his example, pills in the UI communicate those capabilities. He also recommends proactive suggestions based on context. If a scene is already open, the system can inspect it and suggest useful actions the user might take. This gives users a clearer sense of the system's practical boundaries instead of asking them to infer those boundaries from trial and error.
Observability must include activity, provenance, and debugging data
The interface should stream the agent's activity so users can understand what it is doing. Dibia calls for user-facing logs, provenance, and debugging tools. Useful details include the number of tokens used and the time taken by each operation. In BlenderLM, the streamed plan, tool calls, scene updates, and verification steps let users follow the system's progress. These records also give developers evidence when a task fails. A final answer alone does not explain whether the system made a bad plan, called the wrong tool, misunderstood the scene, or stopped after an error.
Interruptibility and cost-aware delegation keep autonomy under human control
Long-running agents need controls that let people pause, resume, checkpoint, and roll back work. A user may notice that the system is taking the wrong route or consuming resources before the task is complete. Dibia also argues that agents need a module that estimates the cost or risk of each action. From the language model's perspective, tool calls are equal unless the system adds this judgment. In Blender, generated Python could perform an extremely destructive operation, so the system should inspect the proposed action and ask the user to take over when the estimated cost is too high. Autonomy therefore includes a decision about when not to continue automatically.
"Your agent is always only as good as the tools you give it, so spend a lot of time, about 50% of your time, on tools."11:37
Who should watch
You are building an agent interface and need concrete patterns for showing plans, tool calls, progress, and verification.
Your system takes actions with side effects, and you need pause, rollback, checkpointing, or human approval controls.
You are deciding whether a task needs multiple agents and want a development process that starts with a baseline and task-specific evaluations.
