A canvas agent can create and edit structured objects such as shapes, diagrams, and forms instead of producing a flat image.
2
Putting agents directly on the canvas lets people see their state, actions, and relationships while they work.
3
Multiple agents can share canvas state, elect a leader, divide tasks, and coordinate on larger pieces of work, although giving agents broader computer access creates serious safety risks.
Summary
Steve Ruiz describes tldraw's experiments with AI on an infinite canvas. Make Real showed how a drawing could become a working prototype, while later experiments let an agent create structured canvas objects through tools and an iterative review loop. Ruiz found that a sidebar agent still felt like handing the keyboard to someone else. Fairydraw moved the agents onto the canvas as visible fairies. Users could watch several agents act at once, ask them to work on separate objects, or have a leader inspect the canvas, create a to-do list, and delegate tasks. The later desktop experiment exposed more of tldraw to agents through an Electron wrapper and an HTTP endpoint that executed JavaScript. That enabled code-to-diagram and interactive canvas experiments, but it also made the safety tradeoff explicit. Ruiz argues that local, file-based apps may be the practical place for this level of agent access because users can choose to accept the risk.
tldraw's canvas can be used as an AI building surface
Ruiz introduces tldraw as an online whiteboard, a startup based in London, and an SDK for building other products. The canvas is built from React components, which gives applications a runtime they can make programmable. He points to Replit's agent canvas and Luma AI's canvas as products built with tldraw's canvas. This matters for the AI experiments because an agent can act on the same objects and components that a person uses, rather than only generating an image or text description.
Make Real turned a sketch into a working prototype
Make Real let a person draw an interface on the canvas, send it to a model, and receive a working prototype. Ruiz says this was especially notable in 2023, before Lovable and before the phrase "vibe coding" had been coined. The user could annotate the generated result and ask for changes such as making parts green, red, or black. Those annotations became part of a prompt that included the earlier website. The demo was unreliable, but it showed how a canvas could connect visual instructions with generated HTML.
Structured canvas objects create different problems from image generation
The AI experiments moved from generating prototypes to creating canvas objects such as circles, shapes, diagrams, and drawings. Ruiz says the model uses text structured outputs to make the same kinds of things a person can make in tldraw. He describes conflicting conventions that make this difficult. On a Cartesian graph, the y-value rises as you move up, while in a web coordinate system the y-value rises as you move down from the top-left corner. Prompting the model to behave predictably required dealing with these conflicts between language, images, and coordinate systems.
An agent loop makes the model review and revise its canvas work
The first canvas collaborator could respond to requests such as drawing a cat, making a diagram, or finishing a graph, but Ruiz felt that one-shot generation did not go far enough. The next version used an agentic loop similar to coding agents. It produced an output, reviewed that output, and iterated until it judged the work complete. The demo asked the agent to draw the life cycle of a butterfly. Ruiz also mentions features such as rejection and showing the agent's thinking and work. Even with this loop, the interaction still felt like giving the keyboard to another AI.
Fairydraw puts visible agents beside the work they are changing
Fairydraw moved the agent out of a sidebar and onto the canvas as a small fairy that users could move around. The fairy's visible state lets people see what an agent is doing, while its position shows where it is acting relative to other agents. Ruiz demonstrates separate requests, such as asking one fairy to draw a hat on a cat and another to draw the cat's neck. The agents can see each other's work and change different parts of the same canvas at the same time. The experiment is available at fairies.tldraw.com.
A leader agent can inspect the canvas and delegate work
When Ruiz selects Helen and Joan along with the other fairies and asks them to draw more animals, one agent becomes the leader. The leader scouts the canvas, creates a to-do list, and delegates tasks to the other agents. Ruiz says the team was working through problems that also appeared in agent orchestration: giving agents shared state, coordinating leader and follower roles, and preventing agents from overlapping in their work. The leader does not perform all the drawing itself. It observes, judges whether the result is finished, and checks whether it was done correctly.
More agent access makes local desktop software attractive
The next experiment gave agents broader access to the canvas through a desktop app. Ruiz wrapped tldraw in Electron and opened an HTTP endpoint that accepted JavaScript and ran it. He calls this a terrible idea for a normal application, but says the risk is different for an offline, file-based desktop app because the user is mainly putting their own machine at risk. The setup allowed agents to visualize code, update code to match a diagram, and add interactivity to canvas drawings even though tldraw did not have built-in primitives for events such as hover or click.
Agent freedom also means accepting sharp safety risks
Ruiz shows that an agent with access to the desktop can do much more than edit a drawing. He describes a team member asking Claude to remove podcasts from Spotify, leading the agent to inspect minified application code and modify it. The agent is willing to change software on the computer without much hesitation. Ruiz says this level of access may be the only way to maximize what an agent can do, but it requires handing control to the user and accepting the risk. His closing view is direct: these are sharp tools.
"The basic idea with Make Real was that you could use a canvas, draw this and send that to a model and have it make it into a working prototype."02:11
Who should watch
You are building an AI feature on a whiteboard or design canvas and need agents to edit structured objects rather than return images.
You are exploring multi-agent collaboration and want a concrete example of shared canvas state, leader-follower delegation, and visible agent activity.
You are deciding how much access an agent should have to a local application, especially when scripting and browser or desktop access create safety concerns.
