Local document conversion to Markdown avoids spending cloud-model tokens before a user asks a question.
2
Heading-based chunks make retrieved answers easier to reference and debug than large, arbitrary document chunks.
3
A database-first pipeline can combine semantic search, BM25 keyword search, telemetry, and code-based guardrails without a large agent framework.
Summary
Abed Matini presents a local-first FAQ chatbot for an employee handbook. The application converts PDFs, Word files, PowerPoint files, and images into Markdown before chunking and indexing them in PostgreSQL. He compares heading, paragraph, fixed-character, sentence, and image-based ingestion strategies, with heading chunks producing clearer references for the demo. Retrieval combines vector similarity with BM25 keyword matching, then uses ranking and a configurable result count before sending context to a local language model. Matini keeps deterministic operations in Python functions, including date lookup, calculations, medical escalation, and prompt-injection checks. The system runs with FastAPI, React, Docker, Ollama, local models, PostgreSQL, and Langfuse. The talk is practical rather than framework-driven. Its main argument is that document preparation, retrieval controls, and pre-LLM validation determine much of a RAG system's cost and behavior.
Local Markdown conversion avoids paying for document tokens before retrieval
Matini begins with the cost and visibility problems of uploading documents directly to a cloud chatbot. The upload consumes tokens before anyone asks a question, and the developer cannot easily see how the service parsed tables, layout, or chunks. His alternative runs Docling locally on the CPU, exports source files to Markdown, and stores the resulting structure under the application's control. The same pipeline can later send only selected context to an online model, or run entirely locally at no model-call cost. He argues that the savings become significant when documents are hundreds of pages long or many employees repeatedly upload material.
Heading-based chunks give FAQ answers a traceable source
The demo compares a whole handbook with a cleaner FAQ file. Large uploads produce fragments such as acknowledgements and signature dates that have little value for answering questions. Matini recommends dividing content into relevant files or question-and-answer entries where possible. His heading-based strategy uses headings detected by Docling as chunk boundaries, so each question and its answer become a retrievable unit. In the demo, a question about employee premiums returns the answer and its source chunk. That makes it easier to see why a result was selected and investigate a failed retrieval. The large handbook chunk returns information, but its origin is harder to track.
Chunking strategies should match the shape and urgency of the source
The application offers several ingestion choices. Paragraph chunking stores each paragraph independently, while fixed-size chunking uses 512 characters with a 64-character overlap in the demonstration. Matini shows that fixed boundaries can split an idea across chunks, although the method is useful for unstructured material that cannot be cleaned first. Sentence-based grouping is another option. For a temporary maintenance email received as a screenshot, the system converts the image to text and Markdown, then indexes sentence groups without requiring manual cleanup. The resulting answer identifies the maintenance window and cites the uploaded image. The interface lets an administrator choose a strategy, inspect chunks, archive files, and delete them.
Python functions keep predictable operations out of extra agents
Matini uses PostgreSQL and calls several ordinary Python functions agents, rather than asking another language model to perform every operation. He prefers this approach for a local system because multiple agent loops would add waiting time and reduce user interest. A function can return the current date or perform a calculation without an LLM, and a test suite can cover its expected cases. The settings page lets him switch agent mode on or off. Agent mode can add searches or product comparisons, but it takes longer and gives the developer less control over references. Direct RAG follows a fixed path from query embedding through hybrid retrieval to the answer.
Hybrid retrieval preserves both meaning and exact terms
The database stores chunks with identifiers, sections, and embeddings. Semantic search finds nearby meanings through vector similarity, while BM25 finds exact words. Matini says both are needed for product names, SKUs, brands, numbers, and medication names, where a merely similar result can be wrong. The system can also filter by identifiers or exact terms before ranking results. Reciprocal Rank Fusion combines the vector and keyword result lists, after which the application returns a configurable number of chunks. His demo uses two for the FAQ, but product search may need more. Medical use cases should return fewer, more focused results because the cost of irrelevant information is higher.
Telemetry makes model use and retrieval behavior visible
The application integrates Langfuse to record each conversation and session. In the demo, a trace shows the anonymous session identifier, the question about weekend maintenance, the model used, the two returned chunks, the references, and the elapsed time in milliseconds. With external models, the same data can support an estimate of spending. Login information can connect activity to different users and sessions. Matini describes this as a way to see how people use the chat and inspect what happened when an answer is poor. The interface also includes a client-side widget, built in React, that can display session information and manage consent.
Code-based guardrails can stop unsafe requests before model generation
Matini places several checks before the request reaches the LLM. A medical question such as how to treat flu can trigger a predefined escalation message instead of generated advice. He also describes prompt-injection checks based on intent rules, term dictionaries, and an LLM classifier, with the goal of rejecting suspicious Markdown before persistence or generation. The system prompt stays small because many instructions and restrictions live in Python code. That makes the behavior easier to test: developers can specify what should be blocked, why it should be blocked, and whether the test passes. He is clear that the dictionaries and sentence patterns need to grow with the product.
A small local model can work when the retrieved context is controlled
Matini first tried a 7-billion-parameter model, then changed the demo to a 0.5-billion-parameter Qwen 2.5 instruct model because it was faster on a smaller server. He says the largest model is unnecessary when the application vets and narrows its data before generation. In his setup, the smaller model is less likely to invent information and can say it does not have an answer when the context is missing. The stack also uses a local BGE embedding model. With Docker, Python dependencies, the React front end, Ollama, and Langfuse running on their own ports, the prototype can be started as a reproducible local application.
"The issue with that is as soon as we upload these documents, we're going to basically lose some of our tokens that we supposed to have only for processing these documents without even asking a question."01:36
Who should watch
You are building a document chatbot and need to inspect how files become chunks before they reach retrieval.
Your RAG application must handle exact product terms, SKUs, numbers, or medical names alongside semantic matches.
You want local telemetry and pre-generation safety checks without adding several LLM-powered agents.
