Traditional multimodal RAG separates images, tables, and text before embedding them, which can lose relationships between elements on a document page.
2
ColPali treats every document page as an image, creates patch-level vectors, and uses late interaction to rank the pages that match a text query.
3
An agent built with Strands can combine ColPali retrieval, a multimodal model, and the built-in speak tool to return a voice response.
Summary
Suman Debnath explains a vision-based retrieval approach for documents that contain mixed text, images, tables, or mostly visual content. Traditional multimodal RAG extracts these elements separately, embeds them, and later reconstructs their relationships. ColPali avoids that first split by treating each PDF page as an image. It divides each page into patches, creates one vector per patch, and compares query-token vectors with patch vectors during retrieval. A late-interaction score ranks whole pages, which can then be passed to a multimodal language model. The workshop demonstrates this flow with ColPali 1.3, Qdrant, a local Docker setup, and a science textbook. Debnath then wraps retrieval in an agent made with AWS's Strands framework and adds image reading and voice output. He is clear that ColPali is not a replacement for OCR-based systems. It is useful when visual structure matters, while traditional methods remain simpler and cheaper for ordinary text.
Traditional multimodal RAG separates document elements before retrieval
Debnath describes a common multimodal RAG pipeline for data containing images, text, and tables. A preprocessing step extracts the three types separately and may attach metadata such as the page associated with an image. A multimodal embedding model then creates vectors for each entity, which are stored in a vector database. At query time, the text question is embedded, relevant chunks are retrieved, and those chunks are sent with the question to a multimodal language model. He also describes variants that summarize each entity into text before embedding, with one version mapping summaries back to the original data.
Separating page elements can lose the relationships that make a document understandable
Some PDFs are made from scanned images, forms, or other visual material, so extracting images, tables, and text does not always preserve useful context. Debnath compares this with IKEA assembly instructions, where a sequence may contain human figures and diagrams without explanatory text. A system that sees only extracted text cannot understand the instructions well. He says ColPali is an option when the document's meaning depends on the visual page as a whole. If existing extraction methods work for the data, he recommends keeping them rather than adding unnecessary complexity.
ColPali represents each page as an image made of patches
ColPali treats a PDF with 100 pages as a dataset of 100 images. It creates patches from every page and produces one embedding vector for each patch. Debnath gives a simple example: if a page has 15 patches, it produces 15 vectors, and a ten-page document produces 150 vectors. This approach does not require first extracting separate image, table, and text entities. The page remains the unit that can be retrieved, while its visual layout and embedded content remain available to the later answer-generation model.
Vision-language training brings image and text representations into a shared space
Debnath explains that vision-language models combine a vision encoder with a text-based model. During training, positive image and text pairs are pushed toward similar vectors, while unrelated pairs are pushed farther apart through contrastive learning. ColPali sends image patches through a vision encoder, a linear projection, and a transformer to produce the patch representations. At indexing time, the document pages are processed without a question. At query time, the text question goes through the corresponding model path so it can be compared with the stored patch vectors.
Late interaction scores a page by matching every query token to its best patch
For a page divided into four patches, Debnath illustrates how each query-token vector is compared with every patch vector using a dot product. For each query token, the system keeps the highest similarity score. It then sums those maximum scores to produce the page's score for the question. The same calculation runs over all pages, and a request for the top five returns the five highest-scoring pages. This is called late interaction because the token-level embeddings are stored first and the detailed comparison happens during search.
The workshop implementation uses ColPali 1.3, Qdrant, and page images
The notebook downloads a science textbook, converts each PDF page into an RGB image, and stores metadata such as document ID and page number. ColPali's preprocessor standardizes the image before the model generates its embeddings. Qdrant runs locally in Docker and stores the multi-vector collection, configured with a maximum-based comparator for the late-interaction calculation. A text question is processed through the same model path, and Qdrant returns the most relevant pages. Those pages are then passed to a multimodal model through Ollama or Amazon Bedrock for answer generation.
Strands turns retrieval and image reading into agent tools
Debnath uses AWS's Strands as a lightweight, model-first framework. He describes an agent as a model paired with tools, without extensive scaffolding. In the RAG example, a custom retrieve_from_qdrant function is exposed as a tool with a decorator. The agent also uses an image reader to format retrieved pages for the chosen multimodal model. The agent receives the question, calls retrieval, passes the selected images to the image reader and model, and produces the final answer. Strands can use models from Bedrock, Anthropic, LiteLLM, or Ollama.
Voice output can be added by giving the agent Strands' speak tool
The voice extension adds the speak tool alongside retrieval and image reading. In the demonstration, the agent retrieves pages about trophic levels, generates an answer, and speaks it in a requested female voice. Debnath then removes the system prompt and asks for a male voice, showing that the result is less deterministic because the default speak behavior remains. He points to the tool specification as the more reliable place to define a persona, although voice instructions can also be placed in the system prompt.
ColPali is useful for visual documents, but it is heavier at indexing time
Debnath says ColPali worked well in an insurance use case involving driver-license images and insurance-policy images, although OCR also worked. The main drawback he reports is the model's weight during data injection, when embeddings are created. Query-time retrieval is relatively fast. He recommends starting with the traditional approach because it is simpler, less resource-intensive, and often more cost-effective. ColPali becomes a better fit when a human needs to see the page to understand it, such as a document with images and text layered together or an IKEA-style manual.
"The moment you have your data and you first segregate these three things, it's just like you have a family you just let your kid go somewhere, you go somewhere and your partner goes somewhere else."16:08
Who should watch
You are building RAG over scanned PDFs, forms, manuals, or pages where layout and images carry meaning.
Your current multimodal pipeline splits tables, text, and images apart, and you want to test page-level visual retrieval.
You want a small agent workflow that combines custom retrieval with multimodal answer generation and voice output.
