GEPA improves an agent by proposing new prompt candidates, evaluating them, and combining candidates that perform well.
2
A golden dataset and deterministic evaluators let teams compare prompts before changing a production agent.
3
Logfire managed variables let teams change prompts, models, temperatures, and other typed settings without redeploying services.
Summary
Samuel Colvin presents a practical workflow for improving agents after deployment. He uses Pydantic AI to extract political relationships from Wikipedia pages about UK MPs, then evaluates the agent against a golden dataset. A simple prompt scores 85%, a more detailed prompt reaches about 92%, and GEPA produces a prompt scoring 96.7% on the demonstrated task. Colvin explains that GEPA uses an agent to propose new prompts, tests them, and moves toward better candidates based on evaluation results. He is clear about the limits: prompts can become verbose, optimization can overfit to its test cases, and a prompt tuned for one model may need to be rebuilt for another. The second half demonstrates Logfire managed variables in a FastAPI application. A typed Pydantic model controls instructions, model choice, and token settings, allowing live updates and percentage-based targeting without redeployment. Colvin also recommends collecting implicit feedback from what users do next.
GEPA searches for better prompt values by combining successful candidates
Colvin describes GEPA as an optimization library derived from "genetic Pareto." It optimizes a string, which can be a text prompt or JSON containing several values. The process keeps candidates from the Pareto frontier, meaning the strongest examples found so far, and combines them to produce new candidates. He compares this with breeding race horses: the process uses the best horses rather than adding a very slow one at random. In the workshop, GEPA is applied to a Pydantic AI system prompt, although the same pattern can cover models, tools, or other agent settings.
The demo task exposes how models confuse ancestors with other relations
The workshop analyzes Wikipedia pages for UK MPs and extracts political relationships. The target is an MP's ancestors, such as parents or grandparents, rather than spouses, siblings, children, or other public figures. Colvin says models often struggle once the prompt uses the broad word "relations," and they include people from the same generation or a later generation. His earlier workaround was to extract every relation and remove spouses, children, and other excluded categories afterward. The agent returns structured output, a list of political relations, using Pydantic schemas.
A golden dataset makes prompt comparisons measurable
The evaluation code loads one case for each MP and compares the agent's structured output with a supposedly correct set of relations. Colvin uses a custom evaluator that generates accuracy metrics and assertions. He prefers deterministic checks against a golden dataset when one is available, while noting that LLM-as-a-judge can produce weaker evaluations. In the demonstration, the simple prompt achieves 85% accuracy. A more detailed human-written prompt reaches about 92% on the comparison run. The evaluation interface also lets users inspect individual cases where the prompts differ.
GEPA can improve accuracy while increasing prompt length
The GEPA adapter uses a Pydantic AI agent to propose a new system prompt for another Pydantic AI agent. It evaluates each proposed prompt, feeds the results back into the optimization process, and continues toward a better candidate. Colvin describes the method as relatively crude despite being state of the art: an agent generates a prompt, the prompt is tested, and successful pieces influence later proposals. In the demo, GEPA reaches a 96.7% performance score, above the roughly 92% score from the best manually written prompt. The resulting prompt is also more verbose.
Optimization can overfit when the test data misses important cases
During questions, an attendee reports that the optimized prompt incorrectly excludes aunts and uncles even though they appear in the golden relations. Colvin explains that the optimizer may have seen only a subset of cases where those relations mattered, so it simplified the prompt in a way that overfit its data. He recommends separate training and validation data that cover the full range of cases. Repeated runs can also reduce variance, although they increase cost. A prompt that performs well on a narrow dataset can still fail on cases outside that dataset.
Managed variables move typed agent settings out of deployed code
Colvin demonstrates a FastAPI server whose agent configuration is stored in a Pydantic model with fields for instructions, model, and maximum tokens. The values are pushed to Logfire and then read by the running application. He changes the instruction from replying in French to replying in German, and later switches the model from Anthropic Claude Sonnet 4.5 to OpenAI GPT-4.1. The point is to change behavior without rebuilding or redeploying the service. Logfire also keeps update history and supports targeting different values at chosen percentages, which enables A/B-style experiments.
Production optimization can cover models, tools, and context strategies
Colvin says prompt tuning is only one possible optimization target. Teams can compare models directly, or optimize model choice for a combination of performance, cost, and latency. Other candidates include the compaction strategy, tool registration, code mode, and separate pieces of a system prompt. He describes a Shopify example that combined agent design, model choice, and GEPA to reduce the cost of analyzing websites while improving performance. He also says optimization matters more for private data and internal instructions that models have not seen during training.
Implicit user actions can provide feedback when explicit ratings are ignored
Colvin says users rarely click thumbs-up or thumbs-down controls, so teams should look for feedback already present in the interaction. In a chat application, a user who says "No, I mean X" or immediately asks again provides evidence that the previous response was poor. A user who thanks the agent or leaves may provide weaker evidence that it was acceptable. He compares this with older search ranking signals, where returning quickly to search again suggested that the clicked page did not answer the query. These signals can help build evaluation data from production behavior.
