Canvas agents struggle because models are trained around text, while spatial work requires understanding positions, shapes, and the effects of actions.
2
A canvas harness can give an agent a screenshot, the underlying JSON, a viewport, and tools for moving around and editing objects.
3
Visible agent characters make multi-agent work easier to follow because people can read their state from the canvas instead of inspecting a chat log.
Summary
Max Drake argues that agents need a spatial harness before they can work effectively on a canvas. Coding agents perform well partly because code is text in and text out, while tasks such as alignment require models to understand two-dimensional space and how actions change it. tldraw's experiments start with giving a model both a canvas screenshot and its underlying JSON. The agent then learns to interpret ordinary shapes and manipulate them, such as adding wind and smoke to a mouse blowing out a candle. The agent starter kit adds goals, todos, and viewport movement. Fairies turn agents into visible, multiplayer characters whose appearance communicates identity and state. Drake then shows a dependency graph where each task is a coding agent, followed by a desktop app that lets agents script the editor with JavaScript. The talk argues that the canvas can become a shared place where people and agents inspect work, coordinate tasks, and interact with interfaces outside the canvas.
tldraw exists to keep canvas engineering from consuming the product
Drake introduces tldraw as both a free infinite-canvas whiteboard app and an SDK for other canvas products. The SDK handles selections, arrows, resizing, and the matrix math behind a canvas. Teams building a Miro competitor, slide designer, or agent canvas can use it instead of rebuilding those foundations. Replit's agent canvas is built on tldraw. Drake says the same problem appeared when teams started building LLM products that manipulated objects in space: people had an idea for the product, then got stuck implementing the canvas itself.
Text-trained agents need extra engineering to work in two dimensions
Drake contrasts canvas work with tools such as Claude Code. Coding agents operate in a medium close to their training, with text as input and output. Asking one to align interface elements exposes a different problem. Agents are very bad at understanding two-dimensional space without additional engineering. The first task for tldraw was therefore to teach a model what it was looking at and how its actions would affect the canvas.
A screenshot and canvas data let a model act on ordinary shapes
In the teaching experiment, the model receives a screenshot, JSON, and other information about the canvas. Asked to make a mouse blow out a candle, it uses ordinary shapes to add wind and smoke and places them correctly. Drake stresses that the mouse is not a special object. The system teaches the model how to interpret the visual and structured representations together, then how to produce actions whose effects it can understand.
The starter kit turns a single canvas prompt into an agent loop
The tldraw agent starter kit wraps the earlier canvas interaction in a harness for agentic work, and Drake says the code is MIT licensed. In his example, the agent is asked to find a friend for a cat somewhere else on the canvas and bring the friend over, with a color preference. The agent creates its own todos, changes its viewport to search for the other objects, and works toward the task. Drake compares this with a coding agent searching a codebase to find where something is defined.
Visible agents make their state readable at a glance
Fairies make agents visible as characters on the shared canvas. People can throw them around, change their hats and colors, and distinguish one from another. Drake calls these details important rather than merely decorative because a high-level view becomes difficult when many agents are active. A person can identify which agent is working from its appearance and position instead of opening each conversation.
A group of fairies can plan, assign, and review work
Selecting several fairies creates a group chat. Drake asks the group to prepare a memo about fiscal-year 2025 financial data. One fairy writes a plan and acts as orchestrator, assigning the task to another fairy and waiting for it to finish. The orchestrator is notified when it should review the result. The canvas shows which agent created the task, which one is working, and which one is waiting, while Drake's colleague's agents continue working elsewhere on the same shared canvas.
A dependency graph can turn every task into a coding agent
Drake shows a real tldraw dependency graph that his team used around the launch of Fairies. The graph makes dependencies, upcoming work, and completion status visible. His Tech Tree prototype gives each node a coding agent that can run autonomously. In the demo, agents build parts of a small multimodal application, and Drake can inspect and merge a pull request from the same interface. He can also draw a task on the canvas, assign it to Claude, and run it. Because the app is multiplayer, colleagues can add tasks, edit the graph, and inspect ongoing work.
A desktop canvas can become a scripting layer for other interfaces
The tldraw desktop app exposes its local editor through a server, allowing an agent such as Claude Code to write plain JavaScript against it. Drake describes this as a code mode for turning the editor into a scripting environment. A colleague used the desktop app as a window manager, drawing rectangles and using computer access to move real windows. Another experiment made Pong use desktop windows as part of the game. These examples extend canvas primitives into interfaces that interact with the rest of the computer.
"Part of the reason why these apps are so good and why they work is because the medium in which they're working, writing code, is essentially the medium in which they were trained."04:24
Who should watch
You are building an agent that needs to select, position, resize, or otherwise manipulate objects in two dimensions.
Your product uses a canvas and you want agents to search across it, maintain visible state, or coordinate with other agents.
You are interested in giving coding agents access to a desktop editor or a shared dependency graph rather than keeping them inside a chat window.
