An LLM judge is useful only when its decisions are calibrated against human annotations tied to the application's actual business rules.
2
Specific binary evaluators with written reasoning are easier to optimize than one general evaluator using scores from one to five.
3
GEPA improved the judge on the airline-support example, but the result remained far from reliable human-level agreement and required careful prompt and data work.
Summary
Mahmoud Mabrouk presents a practical workflow for building an LLM-as-a-judge evaluator for a customer-support agent. He starts with application-specific metrics, asks subject-matter experts to label traces, and records the reasons behind their decisions. He then uses GEPA to mutate and merge judge prompts, selecting candidates through a Pareto frontier rather than average score alone. In the airline-support example, the initial judge mostly marked traces compliant because it did not know the policy. Optimization improved validation accuracy from 61% to 74% and reduced that bias, but the judge still fell well short of strong human alignment. Mabrouk is direct about the work involved: the data was small and imperfect, the policy was complex, prompt-refinement templates needed custom changes, and experiments cost hundreds of dollars. The workshop's practical advice is to inspect small runs, understand errors, improve the seed and reflection prompts, then scale the search.
A fast judge creates false confidence when it does not agree with people
Mabrouk opens with an agent in production whose observability dashboard reports acceptable hallucination scores while customers report failures. A generic prompt that asks an LLM to decide whether an output is a hallucination cannot know the application's rules by itself. Human annotation is slow, so it limits how quickly a team can test prompts and ship changes. An LLM judge can speed up that loop only when its results correlate with human decisions. The same applies to online evaluation, where teams need to tell whether production behavior is improving or whether the data distribution has changed. Calibrated judges also support a feedback loop in which traces produce new evaluations and those evaluations guide further application changes.
Metrics should come from the application and its subject-matter experts
The evaluator should measure properties that matter for the use case, rather than defaulting to broad metrics such as hallucination. For the airline customer-support agent, Mabrouk describes four error types found through trace review: policy adherence, response style, information delivery, and incorrect tool calls. He recommends making these separate judges instead of asking one evaluator to assess every failure mode. The metric-design process starts with subject-matter experts reviewing traces, commenting on what worked and what failed, and clustering recurring errors. This produces concrete evaluator policies that reflect the business rules. It also gives the optimization process a more focused target than a vague overall notion of success.
Binary decisions with reasons give GEPA information it can use
Mabrouk recommends replacing one-to-five ratings and percentages with binary outcomes such as policy-compliant or non-compliant. Even two human annotators can struggle to agree on an exact score, which makes a numerical scale harder to calibrate. Each binary label should include reasoning. In the example, the annotation explains that the agent approved a cancellation without first checking whether the reservation met the airline's cancellation rules. That explanation tells the prompt optimizer what rule was missed. A label without the reason would leave GEPA to infer the policy from the conversation and the final verdict alone, which is especially difficult when the support policy contains many interacting conditions.
The dataset needs inspection before prompt optimization starts
Mabrouk calls data collection and annotation the hardest parts of the workflow. The example uses 599 annotated conversation traces from the airline scenario in T-Bench, with 62% compliant and 38% non-compliant examples. He warns that the data is small, complex, imperfectly distributed, and partly AI-generated from assertions in the original benchmark. The notebook uses 480 training traces and 112 validation traces, split by tasks so that redundancies do not cross the two sets. Reviewing the traces and annotations helps determine whether the examples contain enough information for a judge to learn meaningful rules. Poor data can limit the result even when the optimization algorithm is working as designed.
GEPA searches through prompt mutations, merges, and a Pareto frontier
GEPA starts with a seed candidate, samples new candidates, evaluates them, filters the results, and repeats until its compute budget is used. Prompt mutation asks an evaluator to reflect on failures and propose a better prompt. Prompt merging combines useful parts of two candidates. Instead of selecting only the prompt with the best average score, GEPA builds a Pareto frontier: for each evaluation task, it tracks candidates that perform well on that task, then selects a set that covers the test cases. The training frontier can reach full coverage, but GEPA still has to merge the separate task-specific improvements into one prompt. That merging step was difficult in Mabrouk's experiment.
The seed judge and reflection template shape what GEPA can learn
The initial judge tells the model to assume compliance and switch to non-compliance only when it finds a specific reason. Mabrouk found this important because a generic judge otherwise relies on its own biases and cannot discover the policy reliably. He also replaced GEPA's default reflection template with a custom one. The revised template tells the refiner to inspect the trajectory, the judge verdict, and the human ground truth, then add or restructure policy rules and improve clarity. This gives the refiner a direct instruction to learn the application's rules. Mabrouk says the default template did not learn that requirement well enough in this example.
The measured improvement was real, but the judge was still not trustworthy
The optimized rubric increased validation accuracy from 61% to 74% and training accuracy by nine points. The judge became less biased toward compliance: its compliance rate moved from 98% toward 64%, while recall and precision for non-compliant cases improved from their weak starting point. On the training data, the Pareto frontier reached 100% task coverage, yet the single merged prompt did not solve every case. Mabrouk says the final judge remained far from the roughly 95% accuracy that would indicate strong alignment with human judgment. The result demonstrates that GEPA can improve a judge, while also showing that optimization alone does not remove the limits of complex, imperfect data.
Small experiments and model choices matter more than blindly scaling the search
Mabrouk tested smaller or older models such as GPT-4o for both refinement and judging, and found them unsuccessful on the complex airline policy. He tried several other model combinations, with Gemini for reflection and Grok for judging producing the strongest results in his experiments; GPT-4o mini for both was also useful. His debugging process started with small runs, inspection of generated prompts and reasoning, and manual changes to the refinement template before increasing sampling. He also warns about cost because long traces create many input tokens. Larger models may be needed for refinement, while a cheaper judge can reduce the long-term cost of evaluating many production traces.
"It's not an algorithm that you just take and it works from day one unless for toy examples."34:26
Who should watch
You are using an LLM-as-a-judge in offline tests or production monitoring and need to know whether its decisions match your application's rules.
Your team has human-labeled traces but has not turned the failure reasons into focused evaluator prompts.
You are considering GEPA for prompt optimization and want practical guidance on seed prompts, reflection templates, model selection, debugging, and cost.
