Video AI systems usually answer each query from scratch, so they lose continuity across moments, files, cameras, and years.
2
A video memory layer should store time-bounded moments, entities, appearances, relationships, and corpus-level context in a queryable context graph.
3
Video workers can plan tasks, retrieve and inspect evidence, ground claims to timestamps, and return structured outputs within explicit limits.
Summary
James Le argues that most video AI systems have no memory in the system sense. They sample frames, extract transcripts, or place content in a prompt, which loses the spatial and temporal relationships that define events. A larger context window does not provide durable continuity across files, camera angles, episodes, or years. His proposed memory layer treats video as a spatial-temporal volume and stores reusable primitives in a context graph. TwelveLabs combines semantic chunks, the Marengo multimodal embedding encoder, a spatial-temporal context store, and the Pegasus video language model. Le distinguishes search, which returns candidate moments, from memory, which supports timelines, entity tracking, explanations, and corpus-level questions. He also describes a video worker that plans tasks, retrieves evidence, uses inspection tools, validates results, and follows cost and output constraints. Demos cover World Cup analysis, traffic footage, and advertising placement.
Video meaning depends on continuity across a spatial-temporal volume
Le says video should not be treated as a stack of images with a transcript attached. That approximation can work for some tasks, but it removes continuity. Meaning comes from space, time, modalities, and sequence. His spatial-temporal volume includes visual information, speech, sound, motion, OCR, camera changes, scene transitions, metadata, and time. A useful system must preserve relationships across that volume so an application can traverse them later. This becomes difficult at enterprise scale, where teams may have petabytes of footage and millions of moments to index.
Text-first systems fail through context, memory, and reasoning gaps
Le identifies three problems in the existing stack. Sampling frames, extracting transcripts, and placing everything into a prompt loses the spatiotemporal relationships that define an event. Retrieval, vector search, and larger context windows do not provide durable continuity across another file, episode, camera angle, season, or year. Text-first systems also do not naturally reason over motion and causality or build persistent structures for who appeared, what happened, and what changed. His argument is that video intelligence needs a memory layer that decides what to preserve, how to connect it, and how to retrieve it later.
Video memory must preserve temporal, multimodal, dense, ambiguous, and grounded evidence
Five properties make video memory difficult. Events depend on before and after, so an isolated frame can mislead. Video combines modalities, and a transcript can miss a logo while a frame can miss a spoken claim. It is dense, with many shots, people, objects, actions, and locations packed into a few minutes, but useful signal is uneven. It is ambiguous because people and brands appear under changing conditions, while concepts can emerge over time. Finally, enterprise workflows need to point back to the source moment, so the system must preserve temporal spans and multimodal evidence.
The TwelveLabs stack separates representation, storage, and reasoning
TwelveLabs starts with semantic chunks that capture meaningful temporal units. Marengo, its multimodal embedding encoder, turns those spans into representations of spatial-temporal relations. A spatial-temporal context store preserves reusable structures such as moments, entities, and metadata. Pegasus is the video context-aware language model that reasons over the content for summaries, metadata, synthesis, and comparison. The models are exposed through an API so developers can use them as infrastructure. This stack gives the reasoning model a representation and store designed for video rather than only text tokens.
Search returns moments, while memory supports corpus-level reasoning
Search recovers relevant moments from a large video library, but it does not provide continuity. Memory supports questions that require reasoning about entities, timelines, and evidence across a corpus. The output changes from a time-bounded clip to structured knowledge, a timeline, an explanation, or another composable result. Le describes two scaling dimensions. Across time, a system should reason over years of footage without reprocessing the archive for every request. Across space, it should combine evidence from camera angles, livestreams, body cameras, stock cameras, and event feeds.
Le's context graph connects time-bounded moments, entities, appearances, relationships, time spans, metadata, and corpus-level context. Moments and shots are evidence units. Appearances record where and when an entity appears, while entities can be people, brands, places, or concepts. Relationships capture co-occurrence, sequences, causality, and timelines. At the top are themes, patterns, gaps, and coverage across the collection. Different questions traverse different parts of the graph. Search may go directly to a moment, entity work may expand from a person to appearances, and a storyline may follow relationships across time.
Five design rules keep the memory layer reusable and grounded
Le gives five rules. Ingest once and reason many times, so expensive understanding happens up front rather than for every query. Store primitives such as moments, entities, and appearances instead of only storing answers. Ground every claim in a specific timestamp. Let intent shape memory because sports, brand safety, compliance review, and analytics need different information from the same footage. Keep the layer composable and API-first, with structured, grounded metadata that applications can use for search, editing, analytics, and other workflows.
A video worker adds planning, tools, limits, and evaluation
A static model starts fresh and returns a single answer. Le's video worker operates inside a more controlled system. It knows what memory is available, plans the task, retrieves and inspects relevant moments, synthesizes evidence, validates the result, and returns an output that can be evaluated. Its capabilities include memory, task planning, retrieval, expert tools such as zooming and frame comparison, and explicit limits on time, cost, depth, scope, and autonomy. An output contract can require structured data with references and timestamps. Evaluation checks whether retrieval found the right evidence and synthesis preserved it.
The demos apply video memory to sports, security, and advertising
Using Jockey, Le ingests 67 videos from the 2022 World Cup and asks for near misses, excluding actual goals, with an explanation for each. It then asks for dramatic goals, the build-up, the finish, and the players involved. For player tracking, it follows Lionel Messi across the corpus and describes camera framing, including a dribble past a sliding defender. Other demos count and classify vehicles and pedestrians, detect dangerous traffic events, and identify rainy conditions. An advertising example finds potential placement points such as hard cuts, impacts, beats, player appearances, slow-motion hero shots, and Adidas logos.
"My argument is that video intelligence need a memory layer that decide what to preserve, how to connect it, and how to retrieve later."03:28
Who should watch
You are building search, analytics, editing, compliance, or content-assembly software over a large video archive and need answers that point to source moments.
Your application must connect people, events, or themes across multiple files, cameras, episodes, or years instead of returning one matching clip.
You want a practical architecture for video agents with reusable representations, task planning, inspection tools, output schemas, and explicit cost limits.
