Llama 3 fine-tuning can fail when templates add two BOS tokens, when the Llama 3 instruct template is applied to the base model, or when untrained tokens remain in the embeddings.
2
Padding and end-of-sentence tokens must have different IDs, or fine-tuned models can generate forever. Unsloth checks and fixes this setup automatically.
3
Unsloth's notebooks automate model preparation, chat templates, export to Ollama, and longer-context fine-tuning while reducing memory use through 4-bit training and gradient offloading.
Summary
Daniel Han explains how Unsloth found and fixed bugs affecting Gemma, Llama 3, and Phi-3 fine-tuning. The main section covers Llama 3: duplicate BOS tokens can damage training, the base model has untrained special tokens, and the instruct chat template should not be used with that base model. Padding and EOS tokens also need different IDs, or generation may never stop. Han then covers CPU-only conversion for GGUF and llama.cpp, Ollama export, configurable chat templates, LoRA settings, and training parameters. The accompanying notebooks automate several fixes. He also describes Unsloth's 4-bit training, gradient checkpointing with system-RAM offloading, and support for longer context windows. The talk is practical and warning-heavy. Small tokenizer, template, and export choices can break an otherwise sensible fine-tuning run.
Han says duplicate beginning-of-sentence tokens are a common Llama 3 fine-tuning error. If training uses two BOS tokens while inference uses one, the model's template no longer matches what it learned, which lowers inference accuracy. With Hugging Face's apply_chat_template, one usage pattern adds the BOS token and another can add it twice. The template should not add a BOS token when the surrounding code already does. Han says this is also seen in Mistral and Gemma. Unsloth checks for the duplicate and removes the extra token automatically.
The Llama 3 base model contains special tokens that were never trained
The Llama 3 instruct model has trained versions of certain special tokens, while the base model does not. Han names reserved special tokens from 0 to 250, the end-of-turn token, and the start and end header tokens. Some embedding means are zero because the Llama 3 team did not use those tokens in the model. Training with them can produce NaNs in the gradients. One repair is to set their embeddings to the mean of the trained tokens. That mean must exclude the untrained entries, or the average will be wrong. Training the embedding tokens and LM head is another proposed fix.
The instruct chat template is incompatible with Llama 3 base
Han warns that people often apply the Llama 3 instruct chat template while fine-tuning the base model. This can leave untrained special tokens in the run, producing NaNs and breaking the fine-tune. The instruct template should be used with the instruct model itself. For the base model, users need a compatible template and token setup. Han repeats that the embedding and LM head can be trained as a way to remove the NaNs, but the safer approach is to avoid mixing the base checkpoint with the instruct formatting in the first place.
A padding token is masked during cross-entropy loss. If it has the same ID as the end-of-sentence token, the EOS token is masked too, so the model may produce infinite generations. Han points out that Phi-3 has the same pad and EOS IDs in its setup, which creates this fine-tuning risk. Unsloth checks the vocabulary for an unused token and assigns it as padding. If none is available, it adds a new pad token with a distinct name rather than reusing an existing vocabulary entry. Users should inspect both IDs before training.
Model conversion and system prompts can affect the final model
Community reports added several Llama 3 fixes. Han says conversion to GGUF or llama.cpp should use the CPU conversion path because float16 conversion differs between CPU and GPU. llama.cpp also gained a warning for duplicate BOS tokens. He adds that some Llama 3 instruct fine-tunes work better at inference time when a real system prompt is included. In some cases, users thought they had supplied one but had actually omitted it. These checks sit outside the training loop, yet they can change whether the exported model behaves as expected.
Unsloth automates export and longer-context preparation
Unsloth's notebooks can generate an Ollama model file after fine-tuning, provided the Ollama chat template exactly matches the training template. The notebooks cover Alpaca data and CSV uploads. Han also describes continued pre-training, where the LM head and embeddings can be trained with a learning rate five to ten times smaller than the main rate. Unsloth supports four times longer context with only a small slowdown by offloading gradient checkpointing to system RAM. Offloading to disk makes fine-tuning extremely slow, and the RAM transfers need to be non-blocking.
Training settings trade memory, capacity, and overfitting
In the Colab example, max sequence length should match the data. Setting it far above the dataset's length does not create useful long-sequence training. Four-bit loading cuts memory use and is especially important on a free Tesla T4. For LoRA, Han suggests ranks such as 16, 32, 64, or 128, while warning that a rank that is too large can overfit and consume more memory. All linear layers should be included, including q, k, v, down, up, and gate layers. He suggests setting LoRA alpha to the rank or using twice the rank, and recommends increasing gradient accumulation instead of batch size when memory is limited.
"The formula for the effective batch size is batch size times the gradient accumulation."15:45
Who should watch
You are fine-tuning Llama 3, Phi-3, Gemma, or Mistral and need to check tokenizer, special-token, or chat-template settings before training.
You are exporting a fine-tuned model to Ollama or llama.cpp and want the conversion and template details that can change inference behavior.
You are working with limited GPU memory and want practical settings for 4-bit loading, LoRA, gradient accumulation, and longer-context training.
