# Voice agents with Realtime Video Sidney Primas, LemonSlice | AI Engineer World's Fair 2026 | 26:36 Source: https://www.youtube.com/watch?v=z1dqv74SpUs Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/voice-agents-with-realtime-video Published: 2026-08-18 Tags: inference, multimodal, video, voice ## TL;DR - LemonSlice uses a world-model approach focused on humans to generate full-body avatars with movement, object interaction, physics, and facial expression. - Real-time video requires a causal attention mask, one-step denoising, and a system harness that coordinates GPU and CPU work without stutters or queue failures. - The next model will add an emotion engine, while the longer-term plan is an end-to-end EQ model that handles audio, emotional state, speech, and avatar video. ## Summary Sidney Primas describes LemonSlice's attempt to make video avatars indistinguishable from people on a video call. The company recently deployed a full-body Teddy Roosevelt avatar in a replica Oval Office, where visitors could speak with it in real time. LemonSlice builds on video world models because they can produce movement, object interaction, scene physics, and micro-expressions from one underlying model. The difficult engineering work is making that model interactive and continuous. It must only attend to past frames, reduce denoising from roughly 30 steps to one, and avoid error accumulation during hours of generation. Primas says serving costs are about the same as for a voice model, despite the amount of video data. He also argues that the production harness, which manages GPU and CPU threads, queues, buffering, and interrupts, holds much of the product value. The next step is an emotion engine, followed by a single end-to-end EQ layer above a separate reasoning model. ## Key ideas ### LemonSlice is trying to make avatars pass as humans on video calls [00:01](https://www.youtube.com/watch?v=z1dqv74SpUs&t=1s) Sidney Primas defines LemonSlice's goal as breaking the Avatar Turing test. The target is an avatar that is indistinguishable from a human during a video call. Photorealism is only one part of the problem. He also names emotions, object interactions, micro-expressions, and an internal state that makes the avatar feel human. LemonSlice's Microsoft partnership put this idea into a public setting with a Teddy Roosevelt avatar in a replica Oval Office. Visitors could walk in and talk to it in real time. Donald Trump interacted with the avatar at the opening and stayed longer than planned, returning after his first conversation. ### A visual layer may make AI interactions easier for people to follow [04:02](https://www.youtube.com/watch?v=z1dqv74SpUs&t=242s) Primas argues that people often struggle to keep paying attention when they have to read for long periods, and that voice alone can also be hard to follow. His company therefore expects many future interactions between people and AI to include a visual layer. LemonSlice provides that layer rather than the whole agent. Customers bring their own language model and usually their own voice. The avatar can wave, move its hands, change clothes, move between scenes, and respond during the same video call. Primas points to physics in the demo, including moving earrings and water, as details that make the character feel more realistic. ### LemonSlice focuses world models on humans to get physical behavior from one model [05:16](https://www.youtube.com/watch?v=z1dqv74SpUs&t=316s) LemonSlice starts with world models and trains them around humans. Primas says this makes the initial model harder to train and deploy, but gives the system capabilities that are difficult to add through narrower avatar approaches. These include full-body movement, interaction with objects, movement in the scene, micro-expressions, and emotion. A single image can define the avatar's appearance, including photorealistic, Pixar-like, or cartoon styles. The same inference setup can also change the character's clothes and surroundings. The company exposes this through an API for uses such as language learning, AI sales calls, and other applications that already use voice agents. ### Expressive avatars need audio data and encoders that contain emotion [08:53](https://www.youtube.com/watch?v=z1dqv74SpUs&t=533s) For LemonSlice's video diffusion transformer, audio is central to facial expression and emotion. Primas says most existing audio encoders were trained mainly on audiobooks, which tend to be monotone and contain little emotion. Those encoders do not give a video model enough information to produce expressive behavior. LemonSlice therefore spends time collecting suitable audio data and getting the audio embeddings right. The video model also understands physical behavior in the world. The challenge is turning that capability into an interactive system that can respond to a conversation as it happens. ### Causal generation and one-step denoising make video interactive in real time [10:23](https://www.youtube.com/watch?v=z1dqv74SpUs&t=623s) Ordinary video models can look at both past and future latents while generating a sequence. An interactive avatar cannot see future audio or future user input, so LemonSlice trains with an attention mask that allows the model to look only into the past. Primas then describes a second speed problem: video generation often uses around 30 denoising steps to turn noise into a clean result. LemonSlice reduces this to one step. The combination makes generation causal and fast enough for a live conversation. It also creates a difficult long-term problem because every new video block depends on earlier generated blocks. ### Long conversations expose error accumulation in causal video generation [12:02](https://www.youtube.com/watch?v=z1dqv74SpUs&t=722s) When a model can only look backward, it must use video that it generated earlier. Each block contains some error, so later blocks can add more error to the earlier result. Primas calls this error accumulation and says it becomes especially difficult when generation continues for hours without a reset. The Teddy Roosevelt avatar had generated continuously for eight hours, and LemonSlice was building another deployment intended to run for sixteen hours. He says the company developed a method that, to the team's knowledge, differs from other current approaches and allows long videos without noticeable error accumulation, although he does not explain the method. ### The production harness coordinates every stream around the video model [14:34](https://www.youtube.com/watch?v=z1dqv74SpUs&t=874s) Primas says LemonSlice has brought the cost of serving its video models close to the cost of serving a voice model. That matters because the system generates minutes and hours of video rather than a short clip. He calls the result surprising because video carries far more pixels than voice. The other major engineering problem is the model harness, which manages the live data moving through the product. Separate work runs on GPUs and CPUs, with threads, queues, buffering, interrupts, and queue cleanup. These parts must be orchestrated so the video stays real time and does not stutter. Primas expects much of the durable product value in real-time applications to sit in this orchestration layer. ### An emotion engine should choose reactions from audio and planned speech [16:35](https://www.youtube.com/watch?v=z1dqv74SpUs&t=995s) LemonSlice's next focus is an emotional engine that understands more of the conversation and chooses the right response. Primas says a useful reaction needs the right emotion, timing, duration, and action. Current avatars can feel awkward because they do not listen or react emotionally at the right moment. The planned engine predicts an action from the avatar's audio input and the text it is about to say. The underlying model already has the ability to create natural actions and interactions with the environment, but those actions are not yet controllable or deterministic enough for a live conversation. Primas gives a target launch window of one to two months for the next model. ### A separate EQ layer could connect perception, emotion, and avatar output [18:55](https://www.youtube.com/watch?v=z1dqv74SpUs&t=1135s) Primas expects an eventual end-to-end EQ model to take the user's video and audio directly and produce the avatar's video and audio. Inside one model, it would understand audio, decide what to say, maintain an internal emotional state, and generate the response. A separate IQ model would provide tool calling, deep thinking, and other intelligent capabilities. The EQ layer would handle interaction quality while receiving its intelligence from that other system. In the longer term, Primas expects internal state to live in latent space and track emotional state or goals over time, rather than being expressed only as words. He says early papers are already exploring this direction and expects such models within two or three years. ## Notable quotes - "What we mean by this is making an Avatar that is indistinguishable from a human on a video call." (00:01) - "So, when you do inference, it never can see the future because the future doesn't exist because like you haven't given it those inputs yet." (10:43) - "The costs are about the same as a voice model. Which is crazy to me." (13:55) - "And honestly, I feel like over time a lot more of the value of the things we build will be in figuring out the model harness." (15:35) - "I strongly believe that in the end there'll be a single model that is the EQ layer for AI." (19:15) ## Tools & references mentioned - LemonSlice - Microsoft - Teddy Roosevelt - Donald Trump - Stanford School of Medicine ## Who should watch - You are building a voice agent and need to understand what changes when it gains a continuous visual avatar. - You are working on real-time video generation and want practical problems around causal attention, denoising, long-run error, and serving costs. - You are designing production infrastructure for streaming models, especially GPU and CPU scheduling, interrupts, queues, and buffering.