# Evaling Video Slop Maor Bril, Character.ai | AI Engineer World's Fair 2026 | 23:13 Source: https://www.youtube.com/watch?v=b_PmGocP4rc Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/evaling-video-slop Published: 2026-07-25 Tags: agents, evals, multimodal, video ## TL;DR - Video metrics that inspect individual frames cannot tell whether a clip preserves its story, physics, pacing, character identity, or sound synchronization. - Pairwise comparison gives more consistent judgments than asking people or models to assign absolute scores from 1 to 10. - A fast Qwen3-VL judge trained on deliberately broken footage can run close to video generation and catch quality regressions before release. ## Summary Maor Bril explains why evaluating generated video needs different methods from evaluating text or still images. CLIP can check whether a frame matches a prompt, and other metrics can inspect frame-to-frame drift, but they do not tell whether a sequence tells the intended story. A video also has to preserve physics, character identity, pacing, shot continuity, and synchronization between sound and action. Bril's team first combined metrics, an LLM judge, and human annotation, then found that the process was too slow and expensive. They moved to pairwise preference training, comparing video A with video B instead of assigning absolute scores. After an early model rewarded visual gloss over substance, they rebuilt the data with matched real and AI footage and deliberately corrupted examples. The resulting small VLM can score a 15-second video in about three seconds. Bril recommends relative comparison, explicit story axes, and putting evaluation inside the generation loop. ## Key ideas ### Frame metrics cannot judge whether a video tells its intended story [02:02](https://www.youtube.com/watch?v=b_PmGocP4rc&t=122s) Bril says tools such as CLIP are useful for judging a single frame, while LPIPS can help detect drift between frames. They still miss whether the sequence tells the story the creator meant to tell. A walking character might hover instead of walk, a character might change across shots, and the pacing might ignore the time needed to move between places. Audio adds another requirement: a door slam needs to occur at the same moment as the visible slam. ### Human-calibrated LLM judging was accurate enough to repeat but too slow to run everywhere [04:09](https://www.youtube.com/watch?v=b_PmGocP4rc&t=249s) The first evaluation harness combined frame-level metrics with an LLM judge and human annotation. Humans annotated each generated report, and that feedback was fed back into the judge prompt so its results stayed aligned with human preferences. Bril says this approach was very slow and expensive. The team wanted evaluation closer to generation because an error is cheaper to fix before it propagates into a longer video. Catching drift in starting frames or regenerating a bad six-second clip costs less than rebuilding the combined three-, four-, or five-minute result. ### A small VLM can explain video failures fast enough for the generation loop [07:07](https://www.youtube.com/watch?v=b_PmGocP4rc&t=427s) The team distilled its committee of metrics and experts into a smaller vision-language model. The judge should explain why a clip scored poorly, such as an extra limb, broken physics, or audio that is out of sync. Bril says the trained model takes about three seconds to score a 15-second video. A larger model produced better results, but it was significantly slower, so the added quality did not justify the delay for this use case. ### Pairwise preference is more consistent than absolute scoring [09:05](https://www.youtube.com/watch?v=b_PmGocP4rc&t=545s) Bril asks people to assign a video a storytelling score from 1 to 10 and points out that the same clip might receive a six, five, four, or eight from different people. When people compare two clips and choose which tells the better story, agreement is usually higher. The team therefore trained on pairs, asking whether A or B was better rather than predicting an absolute score. Repeated comparisons make it easier to generalize what is better versus worse. ### A first judge rewarded visual vibe instead of the qualities the team wanted to measure [10:11](https://www.youtube.com/watch?v=b_PmGocP4rc&t=611s) The first model was confidently wrong. One still frame came from a clip where the camera did not move for four seconds, yet the model gave its camera work a 9.2. In other examples, it said physics looked great when ghosts hovered and people flew. Bril attributes this to the training data. The model learned to recognize coherent-looking footage and artificial artifacts, so it rewarded the gloss of a clip instead of checking whether the video actually told the intended story. ### Matched real and AI footage helped turn the model into a quality detector [11:43](https://www.youtube.com/watch?v=b_PmGocP4rc&t=703s) To fix the data, Bril paired real footage with AI footage. He had initially avoided this because the model could simply learn to detect AI video instead of video quality. The team kept encoding consistent on both sides so one clip did not contain artificial compression clues. It also annotated both videos with the same method across the same quality axes. Bril says this produced a much better detector. ### Agents can verify and repair their own video outputs [13:24](https://www.youtube.com/watch?v=b_PmGocP4rc&t=804s) The team changed from a complex fixed pipeline to an agentic workflow. A pipeline can work for a narrow use case, but users bring different stories, characters, images, and voices, which causes the process to drift. Agents can use tools to validate the outputs they create, adapt to those changes, verify their own work, and fix problems as they go. Bril's practical advice is to compare rather than score, measure the actual axes that matter, and place evaluation inside the generation loop. ### Sound evaluation can align timing, while lip syncing remains unsolved [15:27](https://www.youtube.com/watch?v=b_PmGocP4rc&t=927s) For sound, Bril describes using Atmos to check that audio quality is high enough and understandable. The model can use the prompt and video together. If the prompt says that a door slammed, it looks for the event in the frames and checks for a sound spike at the same timestamp. It does not identify the sound semantically; it checks whether the timing contains the expected spike. Bril says lip syncing is still an unsolved problem, especially for animated characters whose mouth movements have no direct relation to speech. ## Notable quotes - "The hard part was never how to make video. The hard part was how do we generate good enough video and how do we judge if the video is good enough?" (01:22) - "Don't score, compare." (09:05) - "Videos are stories. Videos are just another way for us to tell stories to others." (14:34) - "Lip syncing is an unsolved problem yet." (16:50) ## Tools & references mentioned - Character.ai - Kling - SeaDance - Veo - Sora - CLIP - LPIPS - Qwen3-VL - Atmos - Judge Judy - Fable ## Who should watch - You are building a video generation product and need a repeatable way to catch bad clips before users see them. - Your current evaluation checks frames or prompt matching, but misses story continuity, physics, pacing, or sound timing. - You have enough video volume that a slow panel of frontier models and human reviewers is becoming too expensive.