# HTML Is All Agents Need James Russo, HeyGen | AI Engineer World's Fair 2026 | 15:13 Source: https://www.youtube.com/watch?v=Cz4v1WHVyZc Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/html-is-all-agents-need Published: 2026-07-21 Tags: agents, code-generation, human-in-the-loop, video ## TL;DR - HTML, CSS, and JavaScript give video agents a language they already understand from their training data. - Hyperframes renders browser-based HTML deterministically by freezing the clock, seeking frame by frame, waiting for assets, and capturing screenshots. - Agents can produce decent videos from one prompt, but strong results still require skills for video taste, iterative craft, and human editing. ## Summary James Russo explains how HeyGen built Hyperframes, an open-source framework that turns HTML into video. The team tried tools including Remotion, but found that teaching agents custom frameworks and formats reduced their creative output. HTML, CSS, and JavaScript worked better because these are the languages represented throughout the web training data of language models. Hyperframes keeps the wrapper around HTML thin, adding only metadata for timing and related information. Its renderer freezes the browser clock, seeks through every frame, waits for assets to load, and captures each frame before encoding an MP4. This keeps browser previews and final videos aligned while allowing browser technologies such as Three.js, SVG, WebGL, and shaders. Russo also describes skills that teach agents video composition and motion rather than framework syntax. Human storyboarding, iteration, and final editing remain part of the process. He is direct that models still struggle with creative work, which motivates a planned code-to-video benchmark. ## Key ideas ### Great launch videos require more than an avatar [01:35](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=95s) HeyGen began with AI avatars, which Russo describes as the A-roll: the narrator, character, or main subject. A finished video also needs B-roll, images, other assets, animation, captions, and music. The company wants agents to handle every layer of that composition rather than only generate the person speaking. This matters because building a product has become easier with coding agents, while explaining and launching that product through video remains difficult. Russo frames the problem as giving agents a canvas broad enough to create the complete video, not just the central performance. ### HTML is a language models already know [02:13](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=133s) Russo argues that HTML, CSS, and JavaScript are the native languages of LLMs because much of their training data comes from scraped webpages built from those technologies. A new DSL or custom JSON format forces the model to learn another way to express visual ideas. He compares that to asking Shakespeare to write a poem in Japanese or Chinese. His conclusion is to let agents produce video in the language they already use for web pages. He also cites Tarik and Andrej Karpathy discussing HTML as a useful visual output format for LLMs. ### Custom video frameworks can reduce an agent's creativity [03:36](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=216s) The team compared video tools by the quality of their results and how friendly they were to agents. After Effects and Premiere Pro produced high-quality creative work but did not give agents much independent creative output. Lottie and Rive offered coding-oriented formats, yet their JSON and custom XML were less natural and less controllable for models. Remotion looked promising, but the team had to teach the agent the framework, its language, and many examples of correct code. Russo says that instruction ultimately took away much of the creativity. HTML produced better results when the model could work directly in its familiar language. ### A thin HTML wrapper worked better than a large framework [05:18](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=318s) HeyGen used Gemini Flash as a design partner because the team wanted to see whether a smaller model could author workable code. They experimented with wrappers around HTML, CSS, and JavaScript, adding context, larger system prompts, and skills. The thinnest wrapper won. It was essentially HTML with a few data attributes that communicate metadata such as timing. Russo says this result gave the team confidence that the format was right. If a smaller model could produce usable code in that format, larger models and coding agents should be able to work with it as well. ### Hyperframes makes browser output deterministic [06:07](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=367s) Browsers load fonts, images, videos, and other assets asynchronously, and their clocks advance on their own schedule. That is useful for webpages but makes it hard to render the same pixels every time. Hyperframes freezes the browser clock and seeks to each point in time, or each video frame. It waits until the page is loaded and ready, captures a screenshot, then moves to the next frame. After collecting the frames, it encodes them into a video. The preview and the final render use the same browser output, so what the browser shows is what the video receives. ### Browser technologies become video primitives [08:48](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=528s) Because Hyperframes renders browser content, anything that can be rendered in a browser can be included in a video. Russo names Three.js, charts, SVGs, shaders, WebGL, WebGPU, and Lottie. The team also finds visual examples on the internet, adapts them to its needs, and places them into videos. This approach gives agents access to a large existing body of web techniques rather than requiring a separate video-specific representation for every effect. The framework's purpose is therefore broader than basic text and image composition. ### Video skills teach taste while leaving syntax alone [09:32](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=572s) Hyperframes skills focus on video composition, motion, and taste because agents already know how to write HTML, CSS, and JavaScript. A website-to-video skill can receive a website, retrieve its assets and information, and use the site's design and branding. The skill then supplies basic motion examples and guidance for producing a video. Russo says the team continually evaluates and improves these skills to raise the quality of a one-shot result. The skills teach agents how to make a better video rather than how to use a proprietary framework. ### Strong results still need storyboards and human editing [11:03](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=663s) A single prompt can produce a decent working video, but Russo says great output requires craft and taste. The Hyperframes workflow starts with the video's narrative, vision, and mission. The team storyboards it frame by frame, adds motion with HTML, CSS, and JavaScript, and combines the pieces into one video. Its studio handles final editing, with humans able to drag and adjust elements as they would in a normal editor. Newly released keyframes let motion designers coordinate movement frame by frame in the studio, with the aim of supporting work usually done in After Effects. ### The team wants a shared code-to-video benchmark [13:17](https://www.youtube.com/watch?v=Cz4v1WHVyZc&t=797s) Russo reports that Hyperframes had rendered more than 1.3 million videos by open-source users in the previous 90 days, with 267,000 creators trying it, about 15,000 videos rendered each day, and 32,000 GitHub stars. He says the project works with coding agents such as Claude Code, Codex, and Cursor when they can write HTML, CSS, and JavaScript. He is also candid that models remain weak at creative work. HeyGen is working on a code-to-video benchmark with LLM labs and video-agent creators to improve the baseline quality of generated videos. ## Notable quotes - "HTML, CSS, and JavaScript are the native languages of LLMs." (02:16) - "So, the same input that is previewed in the browser is also rendered into the video." (08:48) - "However, great output takes craft." (11:03) - "The one honest thing that we're going to say here is that the models still aren't good at creative work." (13:17) ## Tools & references mentioned - HeyGen - Hyperframes - Gemini Flash - After Effects - Premiere Pro - Lottie - Rive - Remotion - Three.js - WebGL - WebGPU - Claude Code - Codex - Cursor - Tarik - Andrej Karpathy - DesignMD - FrameMD ## Who should watch - You are building a coding agent that needs to create motion graphics or launch videos from code. - Your current video format requires agents to learn a custom DSL, JSON schema, or framework before they can produce useful output. - You need browser previews and rendered MP4s to match while retaining access to HTML and web graphics technologies.