Production-ready AI assistants need batch evaluations that expose missing information and hallucinations.
2
BotDojo traces each flow node so developers can inspect prompts, retrieved data, and model outputs while debugging.
3
Support tickets can be turned into synthetic question-and-answer data, checked for coverage, and written back to the vector index.
Summary
Paul Henry presents BotDojo as a platform for moving LLM applications from a quick prototype to production. He starts with a low-code chatbot that retrieves information from a vector database, sends it to an AI model, and returns an answer with citations. BotDojo exposes the execution trace for each node, including the prompt and retrieved data. Henry then runs batches of questions through the chatbot and evaluates the results for missing information and hallucinations. The demo uses support interactions as source material for synthetic data. BotDojo extracts questions and answers, retrieves relevant information, writes documents in the existing style, checks whether each document can answer the question, and adds the result to the vector index. Henry compares an earlier batch with a run that includes the generated data. The live demo ends with the evaluation display turning green, although the presentation is visibly rushed and the result is not explained in detail.
A working vector-search prototype still needs production testing
Paul Henry begins with the gap between a quick prototype and a dependable application. Connecting a vector database to an LLM can take a weekend, but preparing the result for production is much harder. He presents BotDojo as an AI enablement company focused on helping companies deploy AI applications. The demo focuses on a chatbot and combines two parts of that work: synthetic data generation and evaluations. Henry does not present deployment as a single configuration step. His opening point is that teams need a way to test the application repeatedly and improve it before they rely on it with customers.
The chatbot flow makes retrieval and model execution visible
The BotDojo chatbot takes a user question, checks the chat history, retrieves information from a vector database, and sends the resulting context to an AI model. In the low-code editor, Henry opens the prompt sent to the LLM and shows the returned result. The platform also supports JSON Schema output when the model can produce JSON, with examples including Grok and Claude. Each node can be traced individually. That trace shows the exact prompt sent to the model and the exact data returned by the retriever. Henry describes this detail as useful for debugging because developers can inspect what the application actually passed between its components.
Batch evaluations expose missing retrieval data and hallucinations
BotDojo's batches feature runs many questions through a chatbot or AI flow and evaluates the results together. Henry opens a previous batch with five evaluations and points out red results. In this run, the vector database does not contain enough information for some questions. The evaluations also check for hallucinations. He clones the batch, renames it to indicate that it will use generated data, raises the throughput for the demo, and removes a filter that had been excluding the new documents. The same batch structure can also be used to compare different models and other flow settings. This gives the team a repeatable way to check whether a change improves the assistant.
Support conversations can produce targeted synthetic test data
Henry shows a second flow that generates synthetic data from support tickets. It extracts questions and answers from conversations between live support agents and customers, then uses those interactions as test data for the chatbot. The flow finds relevant information in the existing index and writes a document that follows the existing writing style. This approach gives the team examples tied to real support needs instead of producing unrelated sample questions. Henry presents it as a technique that has been working well for BotDojo customers. The generated material is then passed into the chatbot batch so the team can see whether the added documents address the gaps found by evaluation.
Generated documents are checked before entering the vector index
The synthetic-data flow includes an inline evaluation that checks whether each generated document contains enough information to answer its associated question. After that check, a code node handles the write into the vector index. Henry notes that low-code applications can grow into flows with 40,000 boxes, so BotDojo allows developers to add code where the visual nodes are not enough. The demo supports TypeScript and is intended to support Python soon. This flow therefore combines generated content, an information-sufficiency check, and an index write. The evaluation happens before the new material is used in the support chatbot's retrieval step.
The final comparison shows an all-green evaluation run
At the end, Henry returns to the support chatbot and compares the earlier batch with the run that includes generated data. The live demo is rushed, with the batch finishing during the countdown. When the result appears, the evaluation display is all green. Henry says that the performance improved, though he does not give a detailed breakdown of which questions changed or which evaluation scores moved. The demonstration shows the intended loop: run the assistant against a batch, identify gaps, generate documents from support material, add them to the index, and run the batch again. The result is presented as a practical improvement cycle rather than a one-time chatbot setup.
"One key thing is you can pull a trace of each node and see exactly what we sent to the LLM, what came from the retriever, the exact data."01:44
Who should watch
You have a retrieval-based assistant that works in a demo but you cannot tell which questions it fails or why.
Your support team has a large collection of agent-customer conversations that could become evaluation cases or source documents.
You are building LLM flows in a visual editor and need traces, batch tests, or custom code to debug them.
