Agents need a data harness to work with video, sensor, and other physical-world data because ordinary coding-agent assumptions break at large scale.
2
Materializing metadata into queryable datasets makes follow-up questions much cheaper than repeatedly processing raw files.
3
Schemas, checkpoints, reusable datasets, source code, and knowledge-base files let agents recall earlier work instead of recomputing it.
Summary
Dmitry Petrov explains why coding-agent habits fail on large collections of video, sensor data, and robot telemetry. Raw files contain nested structures such as clips, frames, objects, labels, and confidence scores, so a small number of files can expand into millions of records. Storing metadata as separate JSON files creates latency and consistency problems, while a separate database adds another system and language. Petrov presents DataChain's approach, which uses Pydantic schemas, an execution engine, checkpoints, and queryable datasets connected to files in object storage. Agents can process expensive data once, save the results, and answer later questions from metadata. A knowledge base records the dataset description, session context, schema, statistics, dependencies, and source code. This gives agents and teammates a way to reuse previous work. Petrov's argument is practical: on physical data, recomputation is expensive, so the harness must make recall and materialized state easy.
Physical data needs a harness around the language model
Petrov says an LLM is only the brain of an agent working with physical data. The harness also needs to let the agent see data correctly, run computation, touch and verify results, run tests, and remember important datasets and results. He contrasts this with large-data work at Anthropic and OpenAI, where agents operate over structured business data with tables and query engines. Video recordings, sensor data, and robot telemetry are less forgiving because the useful information is buried inside binary files and multimodal structures.
A small file collection can expand into millions of data objects
Video data is nested. A recording may contain clips, clips contain frames, and frames contain objects with types, classes, labels, and confidence values. Petrov compares this expansion to a neutron star: the visible collection can look small while its internal mass is large. He says 2,000 video files can generate millions of internal objects. A collection of about 90 videos in his demonstration produces about 100,000 records, and larger projects can reach millions before adding deeper object-level detail.
Pydantic schemas connect messy files to ordinary analytical code
Petrov recommends Pydantic as a way to define schemas in the same language as the application code. This avoids creating a separate SQL island inside a Python codebase. In the demo, the generated model includes a video file, frame ID, timestamp, class ID, confidence score, and bounding box. File metadata such as path, checksum, e-tag, and size becomes database columns rather than separate JSON files in S3. Queries such as counting detections labelled 'person' can then run against the database.
An unstructured-data engine must connect files, functions, and results
For terabytes of messy data, the harness needs an execution engine that can distribute work and recover after failures. Petrov describes connecting Python functions, Pydantic input and output models, storage files, and a metadata warehouse. A function can process one file into one object or generate multiple objects, which are saved as a dataset or table. The engine can assign the work across a specified number of machines, using the files and schemas to organize parallel processing.
Checkpoints prevent expensive processing from being lost
Processing binary files can take a long time and may include expensive LLM calls. If a bug or API failure stops a run after hundreds of thousands of files, restarting from scratch wastes the completed work. Petrov calls incremental updates and data checkpoints a must-have. After fixing the problem, the system should continue from existing results. When new files arrive in a bucket, a later run should process only those files and update the dataset without recomputing older work.
Materialized metadata makes data tests and questions affordable
Petrov says tests are slower on binary data than in ordinary coding-agent workflows, even though accuracy matters more because a data question usually has one correct answer. Running complex Python over raw files is the slowest and most expensive option. The harness should first build layers of metadata around datasets, using ideas from dimensional and star-schema data modeling. When asked a question, the agent checks whether an existing dataset can answer it with a single SQL-like query. If not, it builds a reusable metadata layer for related questions.
A knowledge base lets agents recall how datasets were produced
A useful dataset needs more than a name. Petrov lists the session context explaining why it was created, a description, storage dependencies, a data preview, the schema, statistics, and the source code. DataChain stores this information in ordinary Markdown files. The source code is especially important because it explains how the data was generated. Linking source files, processing code, and warehouse results creates data lineage that agents and teammates can use instead of repeating the same computation.
The dataset becomes the agent's reusable unit of state
Petrov's stack starts with physical data in object storage, then uses expensive model calls and an execution engine to extract metadata into dataset slices. A knowledge base shares those results. He argues that stronger frontier models do not solve the underlying cost problem because ordinary agent intuition still pushes toward recomputation. A data harness changes the workflow by making prior computation visible and reusable. His closing recommendation is to build these storage, execution, metadata, and memory layers around the agent.
"In order to make agent to work for unstructured data, for physical data, we need to build not only the brain which we already have, right? It's LLM. But we need to make harness data harness to for agents to understand this physical world."01:35
Who should watch
You are building an agent that searches or analyses large collections of video, sensor, telemetry, or other binary files.
Your current workflow reruns expensive perception or LLM processing whenever a user asks a follow-up question.
You need datasets, checkpoints, source code, and lineage that agents and teammates can reuse across projects.
