From Ingestion to Agents: How AI Teams Build on Document Intelligence

Adit Abraham, Reducto22:29 · Sept 2026 · 16K views
Thumbnail for From Ingestion to Agents: How AI Teams Build on Document Intelligence Watch on YouTube
TL;DR
  1. 1

    Document processing for agents needs different tools for layout detection, semantic correction, tables, retrieval, and verification.

  2. 2

    Agents make bad document inputs more damaging because errors compound across multiple decisions and tool calls.

  3. 3

    Teams should evaluate every stage of the document pipeline, including parsing, retrieval, formatting, and end-agent performance.

Summary

Adit Abraham argues that document intelligence is an infrastructure problem for agent systems. PDFs preserve visual layout rather than meaning, so useful processing has to handle merged cells, charts, handwriting, and inconsistent structures. He recommends combining traditional computer vision with vision-language models: small models can detect layout cheaply at scale, while VLMs handle semantic interpretation and difficult cases. His agentic OCR approach applies token-level corrections instead of asking a model to rewrite a page, which avoids silently changing facts. Abraham also describes choosing markdown or HTML based on table complexity, then creating a natural-language table representation for embedding retrieval. Better parsed inputs improved benchmark performance and reduced reasoning-token use. Classification, splitting, agent harnesses, code execution, repeated checks, and evaluations at each pipeline stage complete the system. He expects agents to work through document file systems and choose tools dynamically rather than follow one fixed pipeline.

Key ideas
01:58

Agents make document errors compound across every step

RAG systems could lose one answer when a document was parsed badly. Agent systems make the risk larger because they make multiple decisions, pull from several sources, and create or modify work products. Abraham says enterprise data is scattered across places such as Google Drive and Box, and is often unstructured and multimodal. The problem includes extraction accuracy, retrieval, and the way an agent edits or uses the context. A clean repository is the exception, so document processing affects every later action in the workflow.

05:28

PDFs preserve visual layout instead of the meaning agents need

Abraham describes PDFs as an old format designed to reproduce a printed page. Humans put meaning into visual choices that are hard to recover from raw file structure. Examples include merged cells, line charts, graphs, slides, and messy handwriting. A table may have no clean grid lines, while a financial presentation may communicate through layout and images. He says a benchmark of decisions from PDF contents gives frontier models about 30 percent, which illustrates how difficult the long tail of real documents remains.

07:19

Traditional computer vision and VLMs should handle different parts of a document

Older pipelines used OCR followed by post-processing and worked well when layouts were predictable, such as a fixed tax form. VLMs extend coverage to varied documents and handwritten text because they can interpret pages more like a person. They are expensive and less deterministic at very large scale, though. Abraham recommends small object-detection models for layout work. These models can run on CPUs and identify difficult regions before a VLM handles semantic interpretation.

09:07

Agentic OCR corrects tokens without rewriting the document

Sending OCR output to a VLM with a prompt to rewrite it can introduce new errors. Abraham says a model may see the word total and recompute the numbers in a table, changing a mistake that was actually present in the source. His approach applies targeted token-level edits, such as changing a period to a comma or distinguishing a zero from an O. A first parsing pass is followed by verification and correction, producing a higher-confidence result without asking the model to regenerate the whole page.

10:42

Table representation should change with complexity and retrieval needs

Simple tables can be represented in markdown, while complex tables with merged cells may need HTML to preserve their structure. HTML gives a language model useful structure, but it can become a noisy block of tags for an embedding model. A person may ask, "how did revenue change over time?" without listing the table's values. Abraham recommends creating a natural-language rendering for retrieval while retaining the HTML version for model reasoning. The same table can therefore have different representations for different consumers.

13:15

Better parsed inputs improve model answers and reduce reasoning work

On a benchmark of decisions from PDF contents, Abraham says structured parse results paired with the original PDF improved the performance of models including Gemini, Anthropic models, and OpenAI models. He says GPT 5.5 and Opus outperformed the newer frontier model in that setup when given better inputs. The models also used fewer reasoning tokens because they spent less effort reconstructing the document and more effort producing the answer. This reduced latency and helped them reach correct answers sooner.

14:41

Classification and splitting prevent irrelevant pages from distracting agents

A document pipeline has to understand what kinds of files are present and route them to suitable processing steps. Abraham says passing a large amount of context can reduce quality beyond its token cost. Paper mail packets can run to hundreds of pages and contain interleaved content, so an agent may waste effort sorting the packet instead of extracting information or making a decision. Classification and splitting give the model smaller, relevant inputs and let it focus on the actual task.

16:17

Agent harnesses combine tools and repeated checks for hard extraction tasks

Line charts are difficult because they are images and because the exact data points can disappear in a rough visual interpretation. Abraham describes an agent with a code interpreter that reconstructs a chart as a markdown table, visualizes its output, finds mistakes, and repeats the process. He applies a similar harness to structured extraction, with a parent agent setting validation criteria for subagents. This approach addresses a trade-off in which frontier models have high precision but silently drop rows, while document services preserve more rows with lower precision.

18:57

Evaluations belong at every stage of the pipeline

Abraham recommends using both public or off-the-shelf datasets and production monitoring because real customer data differs from a designed test set. Teams should inspect parsing, retrieval, formatting, and final agent performance separately. Excellent parsing cannot compensate for retrieval that supplies the wrong context. The useful question is therefore not only whether the document was read correctly, but whether each stage helped the agent complete its task.

"The unfortunate thing is whereas in some blanket evals, you might have contents that is really trying to do the work for the model and say exactly what you're looking for, a real world person does not enumerate the values in the table."12:14
Who should watch
  • You are building an agent that reads PDFs, spreadsheets, forms, or scanned documents and need to decide where OCR, computer vision, and VLMs belong.
  • Your retrieval system handles tables or long document packets, but the model often receives noisy structure or irrelevant pages.
  • You need a practical evaluation plan for a multi-stage document pipeline and want to understand why final-answer tests alone are insufficient.