# LLM Quality Optimization Bootcamp

Thierry Moreau & Pedro Torruella, OctoAI | AI Engineer World's Fair 2024 | 53:05

Source: https://www.youtube.com/watch?v=2Wtq2GvUicw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/llm-quality-optimization-bootcamp
Published: 2025-02-08
Tags: evals, fine-tuning, inference, open-models

## TL;DR
- Fine-tuning is most useful after prompt engineering and retrieval augmented generation have failed to reach the required quality for a focused task.
- A 10,000-example PII redaction dataset was used to fine-tune Llama 3 8B for function calling, producing a model that scored 0.97 compared with GPT-4's 0.68 on the demonstrated evaluation.
- Serving the smaller fine-tuned model through OctoAI reduced the stated inference cost from $30 to $0.15 per million generated tokens, a 200x difference.

## Summary
Thierry Moreau presents fine-tuning as one stage in an LLM quality process. Teams should first improve prompts, then add retrieved or tool-provided context. Fine-tuning becomes a good fit when the task is narrow, such as classification, information extraction, formatting, or function calling, and the team has enough high-quality data. The bootcamp demonstrates the full cycle with a PII redaction task based on the PII Masking 200k dataset. The examples are formatted as logged LLM conversations, uploaded to OpenPipe, and used to create a parameter-efficient Llama 3 8B fine-tune. The resulting LoRA is uploaded to OctoAI for inference. On a 100-example evaluation, the fine-tuned model scored 0.97 against GPT-4's 0.68. Moreau also explains the tradeoffs between open and closed models, small and large models, LoRA and full fine-tuning, and managed and self-hosted deployment. He is clear that the process must be repeated as production data changes.

## Key ideas
### Fine-tuning addresses quality and cost problems after earlier techniques have been tried
[01:55](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=115s)
Moreau frames the talk around two problems: rising GenAI bills and prototypes that cannot meet the quality bar for production. Fine-tuning can improve quality, and a smaller model can also reduce serving costs. He places it inside a crawl, walk, run sequence. Prompt engineering comes first, with techniques such as more specific prompts, few-shot examples, and chain-of-thought prompting. Retrieval augmented generation comes next when the model needs current or external information. Fine-tuning is the next step when those methods still do not produce the required result.

### Narrow tasks and high-quality proprietary data make the strongest fine-tuning candidates
[09:54](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=594s)
Moreau recommends fine-tuning when an LLM is being used for a highly specific task. His examples include classification, information extraction, prompt formatting, and function calling. The team also needs enough high-quality data because the model depends on the examples it is trained on. He describes a further reason to explore fine-tuning: a large GenAI bill from OpenAI, a cloud vendor, or another provider. The approach is less compelling as a general solution for every open-ended language task.

### A production fine-tuning process includes data collection, training, deployment, and repeated evaluation
[11:12](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=672s)
The proposed cycle starts by building a dataset, then fine-tuning a model, deploying it to handle production volume, and evaluating its quality. Moreau says this is an iterative process rather than a fire-and-forget operation. Production data will drift and evolve, so teams need to monitor quality, update the dataset, and fine-tune again. The demo is intended to show one complete pass through this cycle so engineers can apply it to their own data and tasks.

### Real human examples are best, but synthetic data can help teams get started
[12:27](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=747s)
Moreau ranks dataset sources by their distance from real production work. The strongest option has real-world prompts and human responses, such as logged customer-service interactions. A middle option uses real prompts with responses generated by a high-quality model such as GPT-4 or Claude. A fully synthetic dataset uses AI-generated prompts and labels. He says datasets on Hugging Face or Kaggle are often synthetic, and the demo uses a dataset from that category. Every dataset should be divided into training and validation data, with a separate test set for final evaluation.

### Small open models can be more practical for specialized tasks than large models
[16:43](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=1003s)
Open models give the team ownership of the resulting weights and freedom to choose where to deploy them. Moreau contrasts an 8-billion-parameter Llama 3 model with a much larger mixture-of-experts model. Large models have broader context windows and stronger reasoning, but they cost more to fine-tune and serve, and may require hardware such as H100 GPUs. His recommendation is to start with a smaller model such as Llama 3 8B or Mistral 7B. A compact model may learn a focused task surprisingly well.

### LoRA reduces the memory and deployment cost of a specialized fine-tune
[17:39](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=1059s)
The bootcamp uses parameter-efficient fine-tuning, which produces a LoRA, rather than full-parameter fine-tuning, which produces a checkpoint. Moreau gives approximate sizes of 50 megabytes for a LoRA and 15 gigabytes for a checkpoint. The smaller representation needs less GPU memory, and multiple LoRAs can run on the same inference GPU. Full fine-tuning has more parameters to adjust and may be better for expensive tasks such as logical reasoning. For classification, labeling, and function calling, he says a LoRA is sufficient for the demonstrated use case.

### The PII redaction demo turns a labeled dataset into a function-calling model
[25:29](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=1529s)
The demo uses the PII Masking 200k dataset, which contains 54 categories of sensitive information across 229 discussion subjects. Each example includes source text, redacted target text, and the PII items with their categories. The model receives a system prompt, a tool specification, and the message to redact. Its response is a call to a redact function containing the sensitive string and its PII type. This design lets the application decide whether to replace the data with a category, remove it, or substitute stable fake values from a database.

### The demonstrated fine-tune improved the measured score while costing much less to serve
[48:18](https://www.youtube.com/watch?v=2Wtq2GvUicw&t=2898s)
For evaluation, the demo compares the model outputs with labeled ground truth and penalizes missed or incorrectly added PII. The test uses a traditional accuracy-style metric rather than an LLM judge. On the displayed run, GPT-4 scored 0.68 on average and the fine-tuned model scored 0.97. Moreau also states that GPT-4 costs up to $30 per million generated tokens, while Llama 3 8B on OctoAI costs $0.15. The fine-tuned model therefore combines the specialized behavior with a substantially lower stated serving cost.

## Notable quotes
- "Fine-tuning is not really a silver bullet for all the problems you're going to face, but when used in the right context for the right problem, it can really deliver results." (02:01)
- "A model is only as good as the data that the model was trained on." (10:30)
- "There is no finish line." (24:41)
- "That's the difference between prototype and production." (47:50)
- "You need to make sure to continuously monitor quality to retune your model as needed." (50:06)

## Tools & references mentioned
- OctoAI
- OpenPipe
- Llama 3
- GPT-4 Turbo
- GPT-4
- Claude
- Mistral 7B
- Axolotl
- vLLM
- TensorRT-LLM
- Hugging Face Text Generation Inference
- Hugging Face
- Kaggle
- Octo Stack
- PII Masking 200k

## Who should watch
- You have a GenAI bill that is growing quickly and need to examine whether a smaller specialized model could handle part of the workload.
- Your prototype works on some examples, but its accuracy or output format is still below the bar for production.
- You want a practical introduction to preparing fine-tuning data, creating a LoRA, deploying it, and evaluating it against a labeled test set.

## Related talks

- [Everything You Need to Know About Fine-tuning and Merging LLMs](https://aietalks.com/talks/everything-you-need-to-know-about-fine-tuning-and-merging-llms) (Maxime Labonne, Liquid AI, 17:52)
- [The GenAI Maturity Curve, or You Probably Don't Need Fine-Tuning](https://aietalks.com/talks/the-genai-maturity-curve-or-you-probably-dont-need-fine-tuning) (Kyle Corbitt, OpenPipe, 18:03)
- [Domain adaptation and fine-tuning for domain-specific LLMs](https://aietalks.com/talks/domain-adaptation-and-fine-tuning-for-domain-specific-llms) (Abi Aryan, 25:09)
- [Low Level Technicals of LLMs](https://aietalks.com/talks/low-level-technicals-of-llms) (Daniel Han, Unsloth, 2:52:26)
- [Finetuning: 500M AI Agents in Production with 2 Engineers](https://aietalks.com/talks/finetuning-500m-ai-agents-in-production-with-2-engineers) (Mustafa Ali, Method Financial & Kyle Corbitt, OpenPipe, 18:44)