# The Future of Knowledge Assistants Jerry Liu, LlamaIndex | AI Engineer World's Fair 2024 | 16:55 Source: https://www.youtube.com/watch?v=zeAyuLc_f3Q Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/the-future-of-knowledge-assistants Published: 2024-07-13 Tags: agents, deployment, multi-agent, rag, tool-use ## TL;DR - A production knowledge assistant needs better parsing, indexing, query planning, tool use, and memory than a basic RAG pipeline provides. - Single agents can use data services as tools, but they struggle when given too many tools or asked to solve an unlimited range of tasks. - Llama Agents treats agents as deployable microservices that communicate through an API and can be orchestrated to handle larger tasks. ## Summary Jerry Liu describes a knowledge assistant that can accept anything from a simple question to a vague research task and return an answer, report, or structured result. He starts with the data layer. Basic PDF parsing can destroy tables and cause hallucinations, so production systems need better parsing, chunking, and indexing. Above that, agentic RAG uses an LLM for query understanding, planning, tool use, and conversational memory instead of making one retrieval call. Liu then explains why multi-agent systems are useful. Specialist agents can focus on narrower tasks, run work in parallel, and use fewer tools with smaller models. His Llama Agents preview packages agents as microservices with a central communication interface, message queues, and explicit or LLM-driven orchestration. The talk connects these pieces into a service architecture for knowledge assistants that can handle multiple requests and move agent workflows out of notebooks. ## Key ideas ### A knowledge assistant must handle tasks beyond simple questions [01:12](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=72s) Liu defines a knowledge assistant as an interface that accepts many kinds of input, including a simple question, a complex question, or a vague research task. Its output might be a short answer, a research report, or structured data. This broader interface matters because enterprise use cases are moving from document extraction and search toward conversational agents that remember prior interactions and perform actions through external services. Liu presents RAG as the starting point for this system, rather than the complete design. The assistant has to understand the task and return the form of result that the task requires. ### Naive RAG fails because its data and control flow are too simple [01:46](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=106s) A basic RAG pipeline usually applies a parser, sentence splitting, chunking, and top-k retrieval. Liu says this setup is quick to build but unsuitable for production. It struggles with broad or complex queries, has no sophisticated way to interact with other services, and is stateless because it has no memory. In his description, simple RAG is effectively a search system built on retrieval methods that have existed for decades. The path forward starts with better data processing and retrieval, then adds agentic query flows and eventually multi-agent task solving. ### Document parsing can reduce hallucinations before retrieval becomes advanced [03:41](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=221s) Liu argues that LLM applications are only as good as their data. Parsing, chunking, and indexing must turn raw unstructured or semi-structured material into a form that an LLM can use. A PDF parser that collapses a financial report's table into mixed text and numbers can destroy the relationships in the document and lead to hallucinations. He uses a Caltrain weekend schedule as an example. LlamaParse preserves the spatial structure well enough for a model to return the correct train time for a column, while a basic PyPDF-style extraction produces many incorrect answers. Better parsing can therefore improve results even before advanced indexing or retrieval is added. ### Agentic RAG uses the LLM to plan how to use data services [06:25](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=385s) Liu describes agentic RAG as a layer that uses an LLM during query understanding and processing, rather than sending the query directly to a vector database. Data services become tools, and the model decides how to use them to solve the task. The basic ingredients include function calling, sequential or DAG-style query planning, and conversation memory. A reasoning loop can repeatedly call functions, while other approaches plan a full DAG or explore a tree of possible outcomes. This design supports comparisons across documents, follow-up questions that depend on user state, and queries that combine unstructured and structured data. ### Single agents become unreliable when their scope and tool set grow too large [08:51](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=531s) Liu is direct about the limits of a single agent. An agent generally cannot solve an infinite set of tasks, and giving one agent a thousand tools causes it to struggle with current model capabilities. He says specialist agents tend to perform better when each one focuses on a particular task and receives a narrower input. Agents also increasingly interact with services that may themselves be other agents. That creates a reason to think about multi-agent systems, where focused components can cooperate instead of forcing one model to understand every task and tool. ### Multi-agent systems trade one overloaded agent for focused, parallel services [09:30](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=570s) Liu gives several reasons to use multiple agents. Focused agents can operate more reliably on a limited class of tasks, then be combined to solve a larger problem. Multiple copies of an agent can process tasks in parallel, which can make the system faster. Each agent can also work with a smaller tool set, around five to ten tools in his example, allowing the system to use a weaker and faster model. He identifies a design choice between letting agents interact through an unconstrained flow and imposing explicit constraints that force an agent to follow a defined path for a given input. ### Llama Agents moves agents from notebook functions into microservices [11:22](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=682s) Liu introduces Llama Agents as an alpha preview built around the idea that each agent should be a separate service. Agents can be written with LlamaIndex or another framework, deployed as services, and reused across tasks. They communicate through a central API or communication interface, with messages passed through a queue. A control plane can coordinate them, drawing inspiration from systems such as Kubernetes. Orchestration can be explicit, with service flows defined in advance, or implicit, with an LLM deciding which work to delegate based on the current state. The stated goal is a deployable architecture that can handle multiple requests. ### A simple RAG workflow can become a set of independently deployed services [13:34](https://www.youtube.com/watch?v=zeAyuLc_f3Q&t=814s) The demo architecture uses a query rewriting service and a default RAG agent that searches and retrieves information. Other services, such as reflection or general tools, can be added. The query rewrite agent receives the original query and produces a revised one. The next agent searches with that query and returns the final response. Liu says the underlying RAG logic is trivial in this example. The point is to show how even a small workflow can be divided into communicating microservices, allowing multiple client requests to run at once and keeping each agent's logic encapsulated. ## Notable quotes - "RAG was just the beginning." (01:30) - "Good data quality is a necessary component of any production grade LLM application." (03:41) - "A single agent generally cannot solve an infinite set of tasks." (08:51) - "The core goal of Llama Agents really is to think about every agent as just like a separate service." (11:22) ## Tools & references mentioned - LlamaIndex - LlamaParse - Llama Cloud - Llama Agents - RAG - Agentic RAG - PyPDF - Kubernetes - Andrew Ng - DeepLearning.AI ## Who should watch - You are building an enterprise RAG system and need to understand where PDF parsing, indexing, memory, and tool use fit before adding more agent behavior. - Your agent has too many tools or tasks, and you are weighing specialist agents, parallel execution, or a multi-agent design. - You have agent logic running in notebooks and want to think through deployment, communication, orchestration, and concurrent requests. ## Related talks - [Building AI Agents that Actually Automate Knowledge Work](https://aietalks.com/talks/building-ai-agents-that-actually-automate-knowledge-work) (Jerry Liu, LlamaIndex, 17:57) - [Effective agent design patterns in production](https://aietalks.com/talks/effective-agent-design-patterns-in-production) (Laurie Voss, LlamaIndex, 15:38) - [On AI and Knowledge](https://aietalks.com/talks/on-ai-and-knowledge) (Pablo Castro, Microsoft, 17:35) - [Stop AI Agent Hallucinations: 5 Techniques + Production Patterns](https://aietalks.com/talks/stop-ai-agent-hallucinations-5-techniques-production-patterns) (Elizabeth Fuentes, AWS, 55:19) - [Practical GraphRAG: Making LLMs Smarter with Knowledge Graphs](https://aietalks.com/talks/practical-graphrag-making-llms-smarter-with-knowledge-graphs) (Michael Hunger, Jesús Barrasa & Stephen Chin, Neo4j, 19:46)