# Building the Document Context Layer for AI Agents

Jerry Liu, LlamaIndex | AI Engineer World's Fair 2026 | 21:04

Source: https://www.youtube.com/watch?v=RQi7x-navxU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-the-document-context-layer-for-ai-agents
Published: 2026-09-23
Tags: context-engineering, multimodal, rag

## TL;DR
- RAG in 2026 is an agent harness paired with a context layer, with retrieval reasoning moving into the agent.
- Document understanding remains difficult because PDFs store glyphs, coordinates, line segments, and cell positions rather than readable structure.
- LlamaIndex combines fast parsing, deeper vision models, semantic storage, and repeatable workflows to support different accuracy, cost, and latency needs.

## Summary
Jerry Liu frames modern RAG as two connected systems: an agent harness and a context layer. Agents now reason about search terms and use tools instead of relying on fixed top-k retrieval, while programs and workflows are increasingly specified in English. The unresolved problem is the large body of organizational knowledge stored in PDFs, Word documents, PowerPoints, and spreadsheets. Liu describes LlamaIndex's document platform through parsing, semantic storage, and repeatable workflows. Parsing is difficult because document formats preserve display instructions rather than human reading structure. He compares heuristic pipelines with one-shot vision-language model processing and argues for a hybrid approach that routes pages to specialized or frontier models. ParseBench measures document understanding across human-verified pages, while LightParse provides a fast Rust-based first pass. The talk ends with document extraction, citations, search tools, and workflows for invoices, claims, and other business records.

## Key ideas
### RAG has shifted from fixed retrieval pipelines to agent reasoning
[01:33](https://www.youtube.com/watch?v=RQi7x-navxU&t=93s)
Early RAG split a corpus into chunks, embedded them, stored them in a vector database, retrieved a fixed number of results, and sent them to an LLM. Liu says modern agents move much of the retrieval complexity into the agent layer. Instead of adding hacks around weak top-k retrieval, an agent can reason about the best keyword or search term, submit a query, inspect the result, and loop. The retrieval tool can remain basic if the agent can formulate better inputs and use the returned context to solve the task.

### Context and program definition are moving upward toward English
[02:59](https://www.youtube.com/watch?v=RQi7x-navxU&t=179s)
Liu describes a change in how people build agent systems. Earlier discussions focused on managing the context window, but newer systems increasingly connect agents to MCP servers, skills, and tasks. Coding agents also let people describe programs through English rather than only through Python or TypeScript. This applies to nontechnical work such as marketing runbooks. He expects agents to move toward longer tasks where a person provides a goal and a scoring rubric, and the agent uses its available context to work toward that outcome.

### Documents remain a large source of context that agents cannot directly read
[05:21](https://www.youtube.com/watch?v=RQi7x-navxU&t=321s)
Liu calls documents universal containers for unstructured context. He points to organizational data in SharePoint, Box, Dropbox, S3, and similar systems, alongside other sources such as web search, MCP servers, Snowflake, and Databricks. He estimates that more than 10 trillion pages of human knowledge are locked in PDFs, PowerPoints, Word documents, and Excel sheets. LlamaIndex's document platform has a parsing layer, a semantic and storage layer for humans and agents, and a workflow layer for repeatable jobs such as invoice processing, KYC, and claims.

### PDFs preserve drawing instructions instead of reading structure
[08:44](https://www.youtube.com/watch?v=RQi7x-navxU&t=524s)
A PDF is designed for display and printing. Text may be stored as individual glyphs with coordinates, while tables may be stored as border line segments and text placed at cell positions. A multi-column PDF also has no guarantee that its stored character sequence matches normal human reading order. Word and PowerPoint files contain more structure, but their bespoke XML includes tags and formatting details that an agent does not need directly. A useful parser must infer document structure, retain relevant metadata, and preserve enough visual information for the agent to understand the page.

### Document parsing needs a hybrid between heuristics and vision models
[10:47](https://www.youtube.com/watch?v=RQi7x-navxU&t=647s)
Heuristic pipelines inspect file structures, group text, and identify elements such as tables and paragraphs. Vision-language models can read visual structure in one pass, but Liu says they may hallucinate on text-only pages, cost a lot, and lack some of the grounding expected from document software. LlamaIndex combines both approaches. Its system optimizes PDF, Word, and PowerPoint engines, routes work between cheaper specialized models and frontier models, and uses fine-tuned document models for elements such as tables and charts. The goal is to balance accuracy and cost at scale.

### ParseBench measures whether models understand documents for agents
[13:36](https://www.youtube.com/watch?v=RQi7x-navxU&t=816s)
Liu describes ParseBench as a public enterprise document benchmark with 2,000 human-verified pages. It measures tables, charts, content faithfulness, and semantic formatting, with an emphasis on whether agents can understand the documents rather than only whether the output is syntactically correct. The benchmark includes about 50 frontier models, open-weight models, and specialized OCR systems. Liu says its results show that document understanding is not solved, especially as document complexity rises. A useful system needs to cover different points on the accuracy and cost curve.

### Document systems have different regimes for accuracy, cost, and latency
[15:08](https://www.youtube.com/watch?v=RQi7x-navxU&t=908s)
Some regulated financial and insurance workflows need accuracy close to 99 or 100 percent because a bad extraction can damage a financial model or trigger a fraud flag. Those users can pay more for deeper reasoning. Indexing a million or more documents per day has a different requirement: the system should be inexpensive, and an agent can inspect the original document later when it needs more detail. Real-time uploads create a third regime. Processing a thousand documents within a minute is difficult for VLM-based systems, so a fast first pass is useful even when deeper visual processing happens later.

### LightParse gives agents a fast first pass before deeper inspection
[16:45](https://www.youtube.com/watch?v=RQi7x-navxU&t=1005s)
Liu presents LightParse as a free, Rust-based open-source parser that produces markdown without using a VLM. An agent can use it when a large batch of PDFs arrives and then call a deeper parser such as LlamaParse when it needs to inspect a particular page. The fast pass scans the documents efficiently. If a page contains a table or chart whose values require closer analysis, the agent can invoke a slower VLM-based tool. Liu describes LightParse as an installable skill designed to plug into agent systems.

## Notable quotes
- "RAG in 2026 basically decomposes into an agent harness plus a context layer." (00:44)
- "Documents are universal containers for unstructured context." (06:35)
- "Document OCR is hard." (08:44)
- "The fastest open-source parser out there." (17:05)
- "You get back granular citations all the way back to the source document for every extracted output." (19:37)

## Tools & references mentioned
- LlamaIndex
- Claude Code
- Claude Code Work
- OpenClaw
- Codex
- MCP
- Snowflake
- Databricks
- SharePoint
- Box
- Dropbox
- Amazon S3
- Microsoft Word
- PowerPoint
- Excel
- PyPDF
- PyMuPDF
- Gemini
- GPT
- Opus
- LlamaParse
- ParseBench
- Hugging Face
- Kaggle
- LightParse
- BM25
- Rust

## Who should watch
- You are building an agent that needs to read PDFs, office files, invoices, claims, or other business documents instead of only plain text.
- Your retrieval system relies on fixed top-k search and you want to understand how an agent can reason about queries and inspect documents in stages.
- You need to choose between high-accuracy extraction, cheap bulk indexing, and low-latency processing for uploaded files.

## Related talks

- [Building AI Agents that Actually Automate Knowledge Work](https://aietalks.com/talks/building-ai-agents-that-actually-automate-knowledge-work) (Jerry Liu, LlamaIndex, 17:57)
- [From Ingestion to Agents: How AI Teams Build on Document Intelligence](https://aietalks.com/talks/from-ingestion-to-agents-how-ai-teams-build-on-document-intelligence) (Adit Abraham, Reducto, 22:29)
- [Structuring the Unstructured](https://aietalks.com/talks/structuring-the-unstructured) (Cedric Clyburn, Red Hat, 20:41)
- [Forget RAG Pipelines: Build Production-Ready Agents in 15 Minutes](https://aietalks.com/talks/forget-rag-pipelines-build-production-ready-agents-in-15-minutes) (Nina Lopatina & Rajiv Shah, Contextual AI, 1:15:43)
- [Building a Smarter AI Agent with Neural RAG](https://aietalks.com/talks/building-a-smarter-ai-agent-with-neural-rag) (Will Bryk, Exa, 18:42)
