# Optimizing LLMs in Insurance with DSPy

Jeronim Morina, AXA | AI Engineer World's Fair 2024 | 19:29

Source: https://www.youtube.com/watch?v=IAdZxqjZ45U
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/optimizing-llms-in-insurance-with-dspy
Published: 2025-02-16
Tags: evals, fine-tuning, observability, prompt-engineering, tracing

## TL;DR
- LLM applications should be treated as engineered systems with clear goals, rather than as prompts that appear to work after manual tinkering.
- A useful DSPy workflow needs domain examples, traces, annotated evaluation data, and a metric before optimization begins.
- DSPy can compile modular programs into different prompts and weight-tuning strategies, but German insurance use cases need custom metrics and evaluators.

## Summary
Jeronim Morina describes AXA Germany's work on an insurance chatbot that helps customers understand terms and conditions. He argues that AI engineers often spend too much time trying tools and adjusting prompts without defining the real problem or measuring results. His team worked with domain experts, collected production examples, inspected generated prompts and traces, and built evaluation data with labels. They then used DSPy to optimize a program split into modules, rather than applying it to one large prompt and hoping for an improvement. DSPy helped organize prompt and weight optimization, but it did not remove the need for engineering judgment. Morina found that built-in metrics and evaluators were aimed largely at English, so German answers required custom metrics. He is candid that DSPy has a steep learning curve. His practical advice is to begin with handwritten prompts and small evaluations, then introduce optimization only after the task, data, and success measure are clear.

## Key ideas
### LLM work needs a real problem before it needs another tool
[00:00](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=0s)
Morina opens by arguing that AI engineers often tinker with tools, chatbots, and QA datasets without caring enough about the problem they need to solve. A language model is not useful on its own. It becomes useful inside a larger system with a defined purpose. He contrasts manual prompt fiddling with engineering work that starts from first principles. For an insurance company, the real work involves customer data, claims, and service operations, so a prompt that merely appears acceptable is not enough. The engineer needs to know what the application should do and how that outcome will be judged.

### Insurance applications expose the cost of vague requirements
[02:41](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=161s)
AXA Germany's data innovation lab works with business units and their data scientists to improve customer service. Morina describes insurance as a data-focused business that is facing more incoming claims and more labor as climate-related problems increase. He uses a customer-facing chatbot as an example. The first step was to define exactly what the chatbot should do with insurance terms and conditions. Domain experts guided the team through the problem space and helped create examples. That collaboration revealed what the production system was expected to deliver before the team tried to optimize it.

### Prompt libraries can hide the behavior they create
[05:34](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=334s)
Morina describes prompt engineering as difficult and error-prone. Engineers may try instructions such as asking a model to output JSON, use Chain of Thought, add tools, or apply guardrails. These techniques can make output more structured or deterministic, but hardcoded templates make the resulting behavior difficult to inspect. He recommends using a man-in-the-middle proxy to see what libraries actually send to the model. His point is practical: an engineer should inspect the prompts and requests produced by a tool instead of assuming that the tool will solve the application problem.

### Tracing and evaluation replace the 'looks good to me' test
[09:54](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=594s)
Small prompt changes caused large changes in the team's results. Adding chaining libraries and more error handling produced a complex, fragile system that was not suitable for production, even though the task seemed achievable. The team began using Arize Phoenix to inspect traces and understand how calls were connected. Morina says that improvement requires measurement. Building evaluation data took months because the team had to scrape production data, prepare clean input-output pairs, and add labels. He also warns about data leakage, where the input already reveals the expected output.

### DSPy needs a measured baseline before it can help
[11:58](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=718s)
Morina warns against starting with DSPy before understanding the data, having enough examples, and defining a baseline. The team had first written prompts, tested whether the task was achievable, and built a complicated manual system. DSPy then offered a way to optimize the program, but it came with a steep learning curve and was not presented as an automatic recommendation for every project. Its value depends on giving it a signal about what should improve. Without a metric or useful examples, adding DSPy does not create a meaningful optimization process.

### DSPy works best when the application is split into modules
[13:10](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=790s)
The team separated the insurance chatbot into modules instead of giving DSPy one large ReAct-style program. A module might cover retrieval, multi-hop question answering, or combining retrieved pieces into a final answer. In DSPy, the program has an initialization part and a forward path, similar to defining components and then describing how data moves through them. Optimizers can tune prompts and weights against a metric. Morina's chatbot metric concerned finding the right answer. The modular structure gives DSPy information about where and how the program can be improved.

### German insurance use cases need custom metrics
[15:40](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=940s)
Morina found that DSPy's built-in answer metrics did not work out of the box for German text. Exact-match and passage-match evaluation was designed around English use cases in his experience. A more specific ReAct workflow also required the team to create its own module and evaluator. For German optimization, the team needed custom evaluators, especially for exact and passage matching. This means that localization affects the evaluation layer, not just the model's input language. A DSPy program cannot be judged well if its metric does not match the language and task.

### Start with small evaluations before using an LLM judge
[16:51](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1011s)
Morina's final advice is to write prompts by hand first and establish whether the task can work at all. He rejects a subjective metric such as looking at ten outputs and deciding they seem fine. Engineers should write basic evaluations with regular expressions or string comparisons, then use more advanced techniques such as an LLM judge after the simpler checks are in place. He also recommends making very small adjustments because the system is already complicated. DSPy requires study through its documentation and learning resources, rather than a quick tutorial.

## Notable quotes
- "We have to reconsider the thing we are doing these days, which is like prompt engineering, and start programming systems again." (00:43)
- "You cannot improve anything which you don't measure." (10:20)
- "Please don't do anything with DSPy or any other tools when you don't have this base setting." (11:58)
- "Please don't overcomplicate things. It's already super complicated." (17:27)

## Tools & references mentioned
- AXA Germany
- DSPy
- OpenAI
- Microsoft Azure
- Mistral AI
- Arize Phoenix
- Chain of Thought
- ReAct
- LangChain
- Instructor
- Conor Shorten
- Hamza Hussein
- Jason Liu
- MITM Proxy

## Who should watch
- You are building an insurance or customer-service LLM application and need a path from domain examples to measurable production behavior.
- Your team keeps changing prompts and adding libraries, but does not have traces, labeled examples, or a clear success metric.
- You are considering DSPy and want to understand its modular programming model and the custom evaluation work it may require.

## Related talks

- [DSPy: The End of Prompt Engineering](https://aietalks.com/talks/dspy-the-end-of-prompt-engineering) (Kevin Madura, AlixPartners, 1:13:13)
- [On Engineering AI Systems that Endure the Bitter Lesson](https://aietalks.com/talks/on-engineering-ai-systems-that-endure-the-bitter-lesson) (Omar Khattab, Databricks, 19:12)
- [The Unreasonable Effectiveness of Separating the Task from the Model](https://aietalks.com/talks/the-unreasonable-effectiveness-of-separating-the-task-from-the-model) (Maxime Rivest, DSPy & Isaac Miller, DSPy; cmpnd, 17:11)
- [How to Construct Domain Specific LLM Evaluation Systems](https://aietalks.com/talks/how-to-construct-domain-specific-llm-evaluation-systems) (Hamel Husain, Independent consultant & Emil Sedgh, Rechat, 18:45)
- [Judging LLMs](https://aietalks.com/talks/judging-llms) (Alex Volkov, Weights & Biases, 18:39)