# Everything You Need to Know About Fine-tuning and Merging LLMs

Maxime Labonne, Liquid AI | AI Engineer World's Fair 2024 | 17:52

Source: https://www.youtube.com/watch?v=uLrOI65XbDw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/everything-you-need-to-know-about-fine-tuning-and-merging-llms
Published: 2024-09-25
Tags: fine-tuning, inference, open-models

## TL;DR
- Fine-tuning should follow prompt engineering only after a strong evaluation framework shows that prompting is not good enough.
- LoRA and QLoRA reduce training costs by updating adapters instead of all model weights, with QLoRA trading some performance for lower VRAM use.
- Model merging can combine fine-tuned models without GPUs, and techniques such as SLERP and TIES often outperform more experimental mixture-of-experts merges.

## Summary
Maxime Labonne explains where fine-tuning fits in the LLM training lifecycle and how to decide whether a project needs it. He recommends starting with prompt engineering, measuring accuracy, cost, and latency, then moving to fine-tuning when the model still falls short and suitable instruction data can be created. He covers supervised fine-tuning, preference alignment with methods such as DPO, and the properties of useful training data: accuracy, diversity, and task complexity. He then compares full fine-tuning, LoRA, and QLoRA, including their trade-offs in performance, speed, and VRAM use. The second half focuses on model merging. Labonne describes SLERP, TIES, passthrough, and mixture-of-experts approaches, with examples including models built from community checkpoints and repeated layers. His practical view is that merging can produce strong models at very low computational cost, while mixture-of-experts merging remains more experimental.

## Key ideas
### Fine-tuning comes after prompt engineering and evaluation
[02:01](https://www.youtube.com/watch?v=uLrOI65XbDw&t=121s)
Labonne recommends starting with prompt engineering when possible. Teams should first build an evaluation framework with metrics such as answer quality, accuracy, cost, and latency. A custom benchmark can fit a niche use case, while open-source benchmarks can cover more general tasks. If prompting is good enough, there is no need to fine-tune. If the model still fails and the team can create instruction pairs, fine-tuning becomes an option. If it cannot create that data, Labonne says the project may need to be reconsidered.

### The LLM lifecycle moves from prediction to instruction following to preference alignment
[00:56](https://www.youtube.com/watch?v=uLrOI65XbDw&t=56s)
Pre-training gives a model large amounts of raw text and trains it to predict the next token. The result is a base model, which tends to autocomplete an instruction instead of answering it. Supervised fine-tuning adds question-and-answer pairs so the model learns to answer and follow instructions. Preference alignment then uses human preferences to shape behavior. Labonne describes the result of this last stage as a chat model.

### Good instruction data must be accurate, varied, and difficult enough
[05:33](https://www.youtube.com/watch?v=uLrOI65XbDw&t=333s)
Labonne says useful supervised fine-tuning samples need factual accuracy, because incorrect information can damage the model's knowledge. They should also cover many topics and writing styles, depending on the use case. Complexity matters because simple question-answer pairs do not force much reasoning. Examples include chain-of-thought tasks, summarization, and requests to explain something to a five-year-old. He recommends starting with open-source datasets, deduplicating them, filtering low-quality samples, and exploring topic clusters before generating more data.

### LoRA and QLoRA reduce the amount of model state that training must update
[08:25](https://www.youtube.com/watch?v=uLrOI65XbDw&t=505s)
Full fine-tuning updates the entire base model and can provide the best performance, but it is expensive. LoRA freezes the pretrained weights and adds trainable adapter matrices to selected layers, so only a subset of parameters is retrained. The whole model still has to be loaded at higher precision. QLoRA quantizes the pretrained model to 4-bit precision before applying the adapter method. This lowers VRAM use, although Labonne says the quantization can reduce performance.

### Learning rate and sequence length shape fine-tuning runs
[14:33](https://www.youtube.com/watch?v=uLrOI65XbDw&t=873s)
Labonne calls the learning rate the most important hyperparameter and says it depends on the model, so it needs experiments. His general advice is to raise it until the loss explodes, then reduce it. The number of epochs should reflect the dataset size. Sequence length affects VRAM use and trades off against batch size, and it does not need to match the full context window of the pretrained model. Batch size should be increased to make better use of available GPUs.

### Model merging reuses fine-tuned checkpoints without GPU training
[10:35](https://www.youtube.com/watch?v=uLrOI65XbDw&t=635s)
Model merging combines the weights of separately fine-tuned models. It can take advantage of checkpoints produced by the open-source community and does not require a GPU. Labonne points to merged models occupying much of the top of an earlier 7B open model leaderboard. He recommends MergeKit, which implements many merging methods. A family tree diagram shows how merged models can themselves be merged into larger combinations.

### SLERP and TIES provide practical ways to combine models
[11:59](https://www.youtube.com/watch?v=uLrOI65XbDw&t=719s)
SLERP, or spherical linear interpolation, merges two models and allows interpolation factors to vary by layer. TIES reduces parameter redundancy by pruning less significant values and rescaling the remaining weights. Unlike SLERP, TIES can combine more than two models. Labonne recommends trying larger combinations with TIES, including seven or eight models, and presents these methods as stronger practical choices than more experimental alternatives when the goal is higher performance.

### Layer repetition and routed model combinations can work despite limited training
[13:26](https://www.youtube.com/watch?v=uLrOI65XbDw&t=806s)
Passthrough merging concatenates layers from different models, or repeats layers from one model in a self-merge. Labonne gives the example of Meta-Llama-3-20B-Instruct, made by repeating six groups of ten layers from Llama 3 70B Instruct without fine-tuning. He says the model was particularly good at creative writing, though also unhinged. He then describes a FrankenMoE approach that combines feed-forward layers from chat, code, roleplay, and math models behind a router initialized with positive-prompt embeddings. This worked well, but he considers it less reliable than SLERP and TIES.

## Notable quotes
- "If it's good enough with just prompt engineering then probably you don't need fine-tuning, the problem is solved, congrats." (02:20)
- "The first one is the accuracy, we want the samples, the outputs, to be factually correct." (05:56)
- "Model merging is the idea that you can take the weights of different fine-tuned models and you can combine them together." (10:35)
- "It doesn't require any GPU, so it's super efficient and it provides excellent results." (10:50)
- "If you want to increase the performance, it's better to use SLERP and TIES instead of mixture of experts because this is a bit more experimental." (16:46)

## Tools & references mentioned
- Liquid AI
- Hugging Face Transformers
- TRL
- Axolotl
- LLaMA-Factory
- DPO
- KTO
- IPO
- LoRA
- QLoRA
- MergeKit
- SLERP
- TIES
- Meta-Llama-3-20B-Instruct
- Llama 3 70B Instruct
- Beyond
- Phixtral
- LLM Course
- LLM AutoEval
- a16z

## Who should watch
- You are deciding whether prompt engineering has reached its limit and need a concrete way to evaluate the case for fine-tuning.
- You need to prepare supervised fine-tuning data and want guidance on filtering, deduplication, diversity, and task complexity.
- You have several fine-tuned checkpoints and want to test model merging before paying for another training run.

## Related talks

- [LLM Quality Optimization Bootcamp](https://aietalks.com/talks/llm-quality-optimization-bootcamp) (Thierry Moreau & Pedro Torruella, OctoAI, 53:05)
- [Low Level Technicals of LLMs](https://aietalks.com/talks/low-level-technicals-of-llms) (Daniel Han, Unsloth, 2:52:26)
- [Domain adaptation and fine-tuning for domain-specific LLMs](https://aietalks.com/talks/domain-adaptation-and-fine-tuning-for-domain-specific-llms) (Abi Aryan, 25:09)
- [The GenAI Maturity Curve, or You Probably Don't Need Fine-Tuning](https://aietalks.com/talks/the-genai-maturity-curve-or-you-probably-dont-need-fine-tuning) (Kyle Corbitt, OpenPipe, 18:03)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)