AI agents use an LLM to plan, call tools, inspect results, and iterate, but they cost more and add latency than simpler approaches.
2
Screenshots of document pages preserve the relationships between text, images, and tables for multimodal retrieval.
3
The workshop builds an agent with MongoDB, Gemini, multimodal embeddings, vector search, and short-term conversational memory.
Summary
Apoorva Joshi introduces AI agents as systems that use an LLM to reason about a task, create a plan, call tools, and refine the plan from the results. She compares agents with simple prompting and retrieval-augmented generation, and warns that agents add cost, latency, and non-determinism. The workshop then focuses on multimodality, especially documents that combine text with images, charts, and tables. Rather than splitting these documents into text chunks and separate visual elements, the proposed pipeline turns each page into a screenshot and embeds it with a vision-language-model-based multimodal embedding model. MongoDB stores the embeddings and image references, while Gemini handles reasoning over retrieved screenshots. The agent can answer document questions, explain charts, and hold multi-turn conversations through session-based short-term memory. The practical portion provides Python notebooks, including an exercise notebook and a completed solution, so participants can implement the components directly.
Agents are appropriate when the workflow is hard to predict
Joshi separates simple prompting, retrieval-augmented generation, and agents. Simple prompting relies on the LLM's parametric knowledge. RAG adds external information, but it still does not handle complex multi-step work or refine its own responses. Agents let the LLM decide the sequence of steps, take actions through tools, inspect the results, and choose what to do next. She recommends agents for complex tasks without a structured workflow, tasks that can tolerate latency and non-deterministic outputs, and applications that need personalization or adaptive behavior over time. Her warning is direct: agents have higher cost and latency, so they should be used only when needed.
An agent combines perception, planning, tools, and memory
The workshop describes four agent components. Perception gathers information from user input or events such as email and Slack messages. Planning and reasoning use an LLM to decide how to solve the task. Tools give the agent external interfaces, from weather and search APIs to vector stores and specialized models. Memory stores and recalls earlier interactions. Joshi divides memory into short-term memory for one conversation and long-term memory across conversations. The lab implements short-term memory. She presents these parts as software analogues of how people gather information, think through problems, act on the world, and use past experience.
ReAct lets the model revise its plan after every tool result
Joshi contrasts planning without feedback with planning that uses feedback. Chain-of-thought prompting asks the model to think through a problem step by step, either with a zero-shot instruction or examples. The lab uses ReAct, short for reasoning and act. The model produces reasoning traces and an action, the agent executes that action, and the model receives an observation from the tool. It then decides what to do next. This loop continues until the model determines that it has a final answer. The agent code, rather than the LLM, executes the function call.
Multimodal agents search mixed data and reason over it
Multimodality means that a model can process, understand, and generate different data types, including text, images, audio, and video. The workshop uses text and images. Joshi points to research papers, financial reports, organizational reports, healthcare documents, graphs, and tables as examples of mixed-media data. A multimodal embedding model turns different data types into searchable embeddings. A multimodal LLM accepts those data types as input and can generate across formats. When a multimodal LLM can search multimodal data and reason over the results with tools, the result is a multimodal agent.
Page screenshots preserve relationships that text chunking can lose
The agent answers questions over documents and explains charts or diagrams, but mixed-media retrieval makes preparation harder. Traditional pipelines extract text, images, and tables separately, chunk the text, summarize the visual elements, and embed the resulting text. Another option embeds text chunks, images, and tables with a multimodal embedding model. Joshi says both approaches inherit chunking problems, including lost context at chunk boundaries, and add several processing steps. The workshop instead converts each document page into a screenshot. This keeps text, images, and tables together so the retrieved unit preserves their relationships.
Vision-language embedding models reduce the modality gap
Joshi explains that many earlier multimodal embedding systems followed the CLIP architecture, with separate networks for text and images. That can create a modality gap, where unrelated items from the same modality are closer together than related text and images. Vision-language-model-based architectures use a shared encoder for both modalities. In her description, this creates a more unified representation and preserves contextual relationships between visual and textual data. The practical pipeline becomes simple: take a screenshot containing text, images, or tables, pass it through a multimodal embedding model, and store the resulting embedding for retrieval.
The retrieval pipeline stores image references beside embeddings
For each document, the lab creates one screenshot per page and stores the screenshots locally. A production system could put them in blob storage such as Amazon S3 or Google Cloud Storage. MongoDB stores the multimodal embeddings together with metadata containing the image paths. The raw screenshots are not stored in the vector database. When a search returns a matching page, the agent uses the reference to load the screenshot from local or blob storage. Joshi says page overlap and metadata such as page numbers can be added when an application needs more continuity between neighboring pages.
The agent passes retrieved screenshots and chat history to Gemini
The workflow uses a multimodal embedding model for retrieval and Gemini 2.0 Flash Experimental as the multimodal reasoning model. The model has a vector search tool and access to earlier interactions. If it calls the tool, the agent executes the search, loads the screenshots referenced by the results, and passes those images along with the original query and conversation history to Gemini. The model can also answer directly without a tool call, such as when the user asks for an image summary. For short-term memory, each query gets a session ID. The agent retrieves that session's history and stores the new query and response after answering.
"So basically given a document containing a combination of text and images you simply take a screenshot of it pass it through a multimodal embedding model and the embedding that you get from that makes this data ready for retrieval."23:51
Who should watch
You are building a RAG system for PDFs, reports, or research papers that mix prose with charts and tables, and want to preserve page-level context.
You want to understand the moving parts of an agent before adopting a framework, including tool execution, ReAct loops, and conversation memory.
A Python-based workshop with a completed solutions notebook suits you better than a purely conceptual introduction.
