Building the Document Context Layer for AI Agents

Jerry Liu, LlamaIndex21:04 · Sept 2026 · 103K views
Thumbnail for Building the Document Context Layer for AI Agents Watch on YouTube
TL;DR
  1. 1

    RAG in 2026 is an agent harness paired with a context layer, with retrieval reasoning moving into the agent.

  2. 2

    Document understanding remains difficult because PDFs store glyphs, coordinates, line segments, and cell positions rather than readable structure.

  3. 3

    LlamaIndex combines fast parsing, deeper vision models, semantic storage, and repeatable workflows to support different accuracy, cost, and latency needs.

Summary

Jerry Liu frames modern RAG as two connected systems: an agent harness and a context layer. Agents now reason about search terms and use tools instead of relying on fixed top-k retrieval, while programs and workflows are increasingly specified in English. The unresolved problem is the large body of organizational knowledge stored in PDFs, Word documents, PowerPoints, and spreadsheets. Liu describes LlamaIndex's document platform through parsing, semantic storage, and repeatable workflows. Parsing is difficult because document formats preserve display instructions rather than human reading structure. He compares heuristic pipelines with one-shot vision-language model processing and argues for a hybrid approach that routes pages to specialized or frontier models. ParseBench measures document understanding across human-verified pages, while LightParse provides a fast Rust-based first pass. The talk ends with document extraction, citations, search tools, and workflows for invoices, claims, and other business records.

Key ideas
01:33

RAG has shifted from fixed retrieval pipelines to agent reasoning

Early RAG split a corpus into chunks, embedded them, stored them in a vector database, retrieved a fixed number of results, and sent them to an LLM. Liu says modern agents move much of the retrieval complexity into the agent layer. Instead of adding hacks around weak top-k retrieval, an agent can reason about the best keyword or search term, submit a query, inspect the result, and loop. The retrieval tool can remain basic if the agent can formulate better inputs and use the returned context to solve the task.

02:59

Context and program definition are moving upward toward English

Liu describes a change in how people build agent systems. Earlier discussions focused on managing the context window, but newer systems increasingly connect agents to MCP servers, skills, and tasks. Coding agents also let people describe programs through English rather than only through Python or TypeScript. This applies to nontechnical work such as marketing runbooks. He expects agents to move toward longer tasks where a person provides a goal and a scoring rubric, and the agent uses its available context to work toward that outcome.

05:21

Documents remain a large source of context that agents cannot directly read

Liu calls documents universal containers for unstructured context. He points to organizational data in SharePoint, Box, Dropbox, S3, and similar systems, alongside other sources such as web search, MCP servers, Snowflake, and Databricks. He estimates that more than 10 trillion pages of human knowledge are locked in PDFs, PowerPoints, Word documents, and Excel sheets. LlamaIndex's document platform has a parsing layer, a semantic and storage layer for humans and agents, and a workflow layer for repeatable jobs such as invoice processing, KYC, and claims.

08:44

PDFs preserve drawing instructions instead of reading structure

A PDF is designed for display and printing. Text may be stored as individual glyphs with coordinates, while tables may be stored as border line segments and text placed at cell positions. A multi-column PDF also has no guarantee that its stored character sequence matches normal human reading order. Word and PowerPoint files contain more structure, but their bespoke XML includes tags and formatting details that an agent does not need directly. A useful parser must infer document structure, retain relevant metadata, and preserve enough visual information for the agent to understand the page.

10:47

Document parsing needs a hybrid between heuristics and vision models

Heuristic pipelines inspect file structures, group text, and identify elements such as tables and paragraphs. Vision-language models can read visual structure in one pass, but Liu says they may hallucinate on text-only pages, cost a lot, and lack some of the grounding expected from document software. LlamaIndex combines both approaches. Its system optimizes PDF, Word, and PowerPoint engines, routes work between cheaper specialized models and frontier models, and uses fine-tuned document models for elements such as tables and charts. The goal is to balance accuracy and cost at scale.

13:36

ParseBench measures whether models understand documents for agents

Liu describes ParseBench as a public enterprise document benchmark with 2,000 human-verified pages. It measures tables, charts, content faithfulness, and semantic formatting, with an emphasis on whether agents can understand the documents rather than only whether the output is syntactically correct. The benchmark includes about 50 frontier models, open-weight models, and specialized OCR systems. Liu says its results show that document understanding is not solved, especially as document complexity rises. A useful system needs to cover different points on the accuracy and cost curve.

15:08

Document systems have different regimes for accuracy, cost, and latency

Some regulated financial and insurance workflows need accuracy close to 99 or 100 percent because a bad extraction can damage a financial model or trigger a fraud flag. Those users can pay more for deeper reasoning. Indexing a million or more documents per day has a different requirement: the system should be inexpensive, and an agent can inspect the original document later when it needs more detail. Real-time uploads create a third regime. Processing a thousand documents within a minute is difficult for VLM-based systems, so a fast first pass is useful even when deeper visual processing happens later.

16:45

LightParse gives agents a fast first pass before deeper inspection

Liu presents LightParse as a free, Rust-based open-source parser that produces markdown without using a VLM. An agent can use it when a large batch of PDFs arrives and then call a deeper parser such as LlamaParse when it needs to inspect a particular page. The fast pass scans the documents efficiently. If a page contains a table or chart whose values require closer analysis, the agent can invoke a slower VLM-based tool. Liu describes LightParse as an installable skill designed to plug into agent systems.

"You get back granular citations all the way back to the source document for every extracted output."19:37
Who should watch
  • You are building an agent that needs to read PDFs, office files, invoices, claims, or other business documents instead of only plain text.
  • Your retrieval system relies on fixed top-k search and you want to understand how an agent can reason about queries and inspect documents in stages.
  • You need to choose between high-accuracy extraction, cheap bulk indexing, and low-latency processing for uploaded files.