Knowledge-work agents need document tools that go beyond retrieval, including search, file lookup, manipulation, and structured queries.
2
Assistive agents help people work through chat, while automation agents run constrained multi-step processes with less human input.
3
Complex PDFs and spreadsheets require preprocessing and specialized tools because a general LLM or naive RAG pipeline can lose their structure.
Summary
Jerry Liu argues that useful knowledge-work agents need two parts: a strong document toolbox and an agent architecture suited to the task. Enterprise information often lives in PDFs, presentations, Word files, and spreadsheets, so agents need connectors, permissions, metadata, parsing, extraction, indexing, and tools for manipulating documents. Retrieval alone is too narrow. Liu separates assistive agents, which help a person research and produce work through a chat interface, from automation agents, which process batches of inputs through a more constrained workflow and return structured results for review or downstream systems. He describes document parsing that combines traditional techniques with LLMs, vision-language models, and validation. He also presents spreadsheet processing that converts irregular Excel files into normalized tables and exposes specialized tools to an agent. Examples include financial due diligence, enterprise search, invoice reconciliation, and technical data-sheet processing.
Liu frames knowledge work around unstructured enterprise data, including PDFs, PowerPoints, Word documents, and Excel files. People have traditionally read these documents, made decisions from them, and then taken actions. Agents can now reason over large amounts of context, perform research, synthesize findings, and act. He separates two broad uses. Assistive agents use a chat-style interface to help a person find information or produce a unit of work. Automation agents handle routine operational tasks in the background, with less human involvement at each step. This distinction determines how much freedom the agent gets and where people review its work.
A document toolbox needs more than vector retrieval
Liu describes a document toolbox as the layer that lets an agent work with enterprise data. It begins with connectors to sources such as SharePoint, Google Drive, S3, and Confluence. Those connectors must carry over permissions and metadata. The system also needs document parsing and extraction that understand tables, charts, and other content, followed by an indexing layer. The available indexes might include vector search, SQL tables, or graph databases. The resulting document MCP server can expose semantic search, file lookup, file manipulation, and structured queries. This gives the agent several ways to work with documents instead of limiting it to one retrieval-and-synthesis step.
Complicated documents contain embedded tables, charts, images, irregular layouts, headers, and footers. They were made for people, so an agent can fail before its reasoning begins if preprocessing loses that structure. Liu describes combining LLMs and vision-language models with traditional parsing techniques. The system also uses validation and reasoning at test time to improve the result. He says that sending a screenshot of a PDF to ChatGPT or Claude is a useful baseline, but it does not provide high accuracy. In his account, document understanding has to be treated as a core part of the agent system, rather than as an afterthought around the language model.
Irregular spreadsheets need semantic structure before an agent can use them
Liu presents an Excel agent for spreadsheets with gaps in rows and columns, where ordinary RAG and text-to-CSV approaches do not fit the data. The system transforms an unnormalized spreadsheet into a normalized two-dimensional format and supports questions over both versions. It first learns a semantic map of the sheet, adapting to the particular document format, then turns that map into specialized tools for an agent. Liu describes reinforcement learning as part of the structure-understanding process. An LLM with a code interpreter is his baseline for spreadsheet work, while specialized tools help the agent reason over the sheet until general code-writing models improve further.
Agent freedom should match the amount of automation required
Liu contrasts constrained and unconstrained orchestration. A ReAct loop, function calling, or CodeAct gives an agent tools and lets it decide how to proceed. This fits assistive interfaces, where a person can guide the agent through a chat conversation. Automation interfaces take batches of inputs and run in the background or on demand. They use more explicit control flow so the process does not go off course. The output is usually structured data, an API action, or a decision that moves downstream after approval. These systems still include people, but review tends to happen after processing rather than at every step.
Automation agents can prepare data for assistant agents
Liu describes a relationship between the two architectures. An automation agent can process and structure large amounts of unstructured data in the background and expose tools over the result. An assistant agent can then use those tools in a user-facing research or search experience. He gives financial due diligence as an example: an automation stage handles public and private financial data in Excel files, PDFs, and PowerPoints, while analysts use a copilot to inspect the structured information and generate reports. The separation lets the background system do repetitive extraction while the person-facing system spends its reasoning budget on questions and analysis.
Real automation encodes business rules and keeps review points
Liu describes technical data-sheet processing for a global electronics company. The agent parses the documents, extracts specified information, matches it against business rules, and writes structured data into SQL. Human review remains part of the process. The purpose is to encode the company-specific logic into an end-to-end workflow instead of asking a writer to handle each document manually. He also names invoice reconciliation, financial data normalization, contract review, and enterprise search as related document-agent applications. The examples make his architecture concrete: the system needs document-specific extraction, explicit rules, a destination for structured results, and a place for approval before downstream action.
"Automation agents can structure process your data and provide the right tool interfaces for assistant agents."14:19
Who should watch
You are building a document agent and need to decide which parsing, indexing, search, and manipulation tools it should expose.
Your product needs to move from a chat assistant toward background processing with structured outputs and human review.
Irregular PDFs or spreadsheets are causing failures even though the language model itself appears capable.
