Prompt improvement can be automated by combining an evaluator with an agent that proposes and tests new prompts.
2
A ground-truth evaluator can score a RAG pipeline by checking whether expected facts appear in its answers.
3
The demo improved its score from 0.4 to 0.9 after two automated iterations, although the result overfit the examples used for evaluation.
Summary
Nir Gazit presents an automated way to improve prompts for a simple Traceloop documentation chatbot. The chatbot uses retrieval-augmented generation with a Chroma database and OpenAI. Instead of manually editing its prompts, Gazit creates 20 questions and defines three expected facts for each answer. An evaluator checks whether those facts appear, returns pass or fail results with reasons, and produces an overall score. A CrewAI agent researches prompting guides, examines evaluator failures, generates a revised prompt, and submits it for another evaluation. In the demo, the score rises from 0.4 to 0.9 after two iterations. Gazit is direct about the limitations. The evaluator overfits the 20 examples because they were all exposed during optimization. He recommends separate training and test sets. He also admits that building the optimizing agent required substantial prompt engineering, so the talk ends with a recursive question about which prompts should optimize the others.
Prompt editing can be replaced by an evaluation loop
Gazit begins with the claim that prompt engineering never really existed as engineering because it usually means asking an LLM to behave properly. He describes a Traceloop documentation chatbot that worked only "kind of okay" after deployment. It needed to answer questions about Traceloop, stay useful to users, and make fewer mistakes. His proposed replacement is an automatically improving machine. An agent researches prompting advice, applies it to the chatbot prompt, runs an evaluator, studies the failures, and repeats the process. The engineer defines what counts as a good answer, then lets the loop search for a prompt that scores well.
The chatbot is a simple RAG application. A question is used to find relevant documents in a Chroma database, then OpenAI receives the question and retrieved context to produce the final answer. Gazit shows the trace with calls to OpenAI and the Chroma database. The pipeline has prompts that could be optimized, but the point is to improve them through the surrounding system. Because the pipeline has separate retrieval and answer-generation stages, its behavior can be examined at several levels instead of treating the final response as an unexplained result.
An evaluator turns answer quality into a measurable score
Gazit chooses an LLM judge because it is easier to build and deploy for this demo. He notes that classic NLP metrics can also evaluate tasks such as translation or summarization, but they generally need a ground-truth answer. His evaluator uses 20 questions about the documentation. For each question, he writes three facts that the generated answer should contain. The judge checks every fact, returns a Boolean pass or fail, and gives a reason when a fact is missing. The system then summarizes the results as a numerical score across 60 expected facts.
Evaluation can target retrieval, generation, or the whole trace
A RAG pipeline can be evaluated at different points. Gazit describes checking whether the vector database retrieved the information needed for a question, which functions like a unit test for retrieval. The complete pipeline can also be judged from the input question and final answer. A deeper evaluation can inspect the question, retrieved context, and answer together. This lets the evaluator ask whether the answer is appropriate given the material the system retrieved, rather than assigning blame to the entire pipeline when one internal step failed.
The agent uses failures and online guidance to rewrite prompts
The optimizing agent starts with an existing prompt and runs the evaluator to obtain an initial score. It then combines the reasons for failed checks with prompting guides that it finds online. From that material, it generates a revised prompt and sends the prompt back through the evaluator. Gazit builds this researcher agent with CrewAI. During the demo, the agent calls the evaluator, receives the score and responses, reasons about why the prompt failed, and generates another version with additional instructions or prompting practices. The cycle resembles classic machine-learning training, although Gazit describes it as training with "a bit of vibes."
Two iterations raise the demo score from 0.4 to 0.9
The initial evaluation score is 0.4. After two iterations, the agent has produced a much longer prompt with instructions about answering Traceloop questions and handling user requests. Gazit stops at 0.9, which in this setup means that 90 percent of the expected facts were correct. He did not manually edit the chatbot prompt during this process. He did write a large amount of code to build the agent, evaluator, and loop. The result shows how an evaluator can provide a score that an automated system tries to improve.
The demo overfits because the optimizer sees every example
Gazit acknowledges that the prompt was optimized against all 20 examples, so it may work well for those questions and fail on a new one. He compares this with classic machine learning and recommends splitting the data into training and test or evaluation sets. The training set is used for optimization, while the separate test set checks whether the prompt generalizes. Giving the optimizer too much context can teach it how to answer the specific questions instead of improving its general behavior. This is the main limit he identifies in the demo.
The agent still depends on manually engineered prompts
Gazit revises his opening claim near the end. He had to do substantial prompt engineering for the agent that optimizes the chatbot's prompts, and he describes that work as difficult. The same issue may apply to the evaluator prompt or the agent prompt. This creates a recursive problem: one agent could potentially optimize the prompts used to optimize other prompts, but Gazit does not claim that this works. His practical conclusion is narrower. Engineers can use evaluators and automated iterations to reduce manual prompt editing, while still needing to design the surrounding system carefully.
"I told you that prompt engineering is dead but I've actually done a lot of prompt engineering for this demo because I needed to engineer the agent that is optimizing my prompts."13:13
Who should watch
You have a RAG application whose answers are inconsistent and you want a repeatable way to improve its prompts.
You are building evaluators and need an example of scoring generated answers against expected facts.
You are considering automated prompt optimization and want to see the overfitting problem before applying it to production.
