The agent writes JavaScript or TypeScript code to edit videos through Diffusion Studio Core in a browser.
2
A visual feedback tool samples the composition at one frame per second and decides whether the agent should render or revise it.
3
The Python agent uses Playwright, browser tools, Chrome DevTools Protocol, and optional remote GPU-accelerated browser sessions.
Summary
Mam presents an open-source video editing agent built through a collaboration between Diffusion Studio and Re-Skill. The project started because Re-Skill needed automatic video editing for its personalized learning platform. The team found limitations in FFmpeg and unreliable server-side rendering with Remotion, then chose Diffusion Studio Core because its JavaScript and TypeScript API renders compositions in the browser. The agent starts a browser session with Playwright and connects to an editing UI built for agents. It generates code from a user prompt, searches documentation through RAG when needed, and sends sampled frames to a multimodal visual feedback tool. The feedback determines whether the composition is ready to render or needs another edit. Mam also describes file transfer through Chrome DevTools Protocol, an llms.txt file for agent documentation, and a planned TypeScript version. Remote browser sessions can provide GPU acceleration and separate sessions for each agent.
The project began with a need for automatic video editing
Mam says Re-Skill needed an automatic editing tool for its personalized learning platform. The team first ran into limitations with FFmpeg. Remotion seemed promising, but its server-side rendering was unreliable. After trying Diffusion Studio Core, they liked its API because it did not require a separate rendering backend. They then met the library's author and decided to build the agent together. This origin matters because the system was shaped by a concrete production need rather than by a generic experiment with language models.
The agent uses code because video compositions already have a programming interface
Diffusion Studio Core supports complex compositions through a JavaScript and TypeScript programming interface. That lets a language model generate code and run it to create or modify a video. Mam argues that code is a strong way to express actions performed by a computer. The design also follows research that, according to Mam, found language-model tool calling in code works better than tool calling in JSON. The agent therefore produces executable editing actions instead of only filling out a fixed data structure.
The editing agent operates inside a browser session
The architecture starts a browser session with Playwright and connects to the operator UI at operator.diffusion.studio. The web application is designed as a video editing interface for AI agents. Diffusion Studio Core renders the video directly in the browser through the WebCodecs API. Helper functions move files between Python and the browser through the Chromium DevTools Protocol. This keeps the editing engine and the agent connected without requiring a separate rendering backend.
Three tools divide editing, documentation search, and visual checking
The agent has a VideoEditingTool, a DocsSearchTool, and a VisualFeedbackTool. VideoEditingTool turns the user's prompt into code and runs that code in the browser. When the agent needs more information, DocsSearchTool uses retrieval-augmented generation to find relevant material from the operator documentation. After each execution step, the composition is sampled at the current rate of one frame per second. Those frames go to VisualFeedbackTool for inspection.
Visual feedback can send the agent back for another edit
Mam compares the visual feedback tool to the generator and discriminator idea from generative adversarial networks. The tool examines the sampled composition and provides a decision about its state. When it gives the agent a green light, the agent proceeds to render the composition. If the result needs work, the feedback loop allows another refinement step before rendering. The system therefore checks the visible output during the process instead of waiting until the final render to inspect it.
Agent documentation is exposed through llms.txt and prompt templates
The team shipped an llms.txt file containing information for agents. Mam compares its role to robots.txt, but for language-model systems. The file gives an agent a machine-readable place to find relevant documentation for the editing UI. Mam also says that combining this information with specific template prompts can take users far in video editing. The documentation search tool uses this kind of context when the generated code needs more information about the available interface.
The browser can move to remote GPU-backed infrastructure
Users can bring their own browser and run the agent locally. The architecture also supports connecting to a remote browser session through WebSocket. Mam says each agent can receive a separate browser session, with GPU acceleration available through the remote setup. A load balancer sits behind those sessions. The proposed remote endpoint is chrome.diffusion.studio, although the remote connection is still described as work in progress.
The first implementation is in Python while TypeScript is in progress
The first version of the agent is written in Python. Mam says a TypeScript implementation is underway. The project combines the Diffusion Studio collaboration with Re-Skill's original need for automated editing. The result is an agent that can generate editing code, run it in a browser, inspect sampled frames, and continue or render based on visual feedback. The talk closes by identifying the implementation language transition rather than presenting the system as finished.
"After each execution step a composition is sampled currently at one frame per second and they are fed to visual feedback tool."03:08
Who should watch
You are building an agent that needs to manipulate video or another browser-based creative tool through generated code.
Your current workflow depends on server-side video rendering and you want to understand the browser-rendering alternative described here.
You need a pattern for combining tool calls, documentation retrieval, visual inspection, and remote browser execution.
