LLM implementations can contain bugs in activations, precision handling, positional embeddings, sliding-window attention, and tokenizers, so engineers should compare implementations against the original model specification.
2
Transformer training becomes practical because the causal attention mask lets the model predict every next token in parallel instead of processing each prefix separately.
3
Unsloth speeds up fine-tuning with custom kernels, lower memory use, careful precision handling, and support for workflows such as QLoRA, CSV data preparation, and model export to Ollama.
Summary
Daniel Han presents a hands-on workshop about reading, testing, and improving language-model implementations. He starts with examples from Gemma, Phi-3, Llama 3, Mistral, and NVIDIA's NeMo model, showing that small implementation differences can change model behavior. The workshop then builds a Transformer from its matrix operations: token embeddings, causal attention, RMSNorm, RoPE, gated MLPs, residual connections, and the language-model head. Han explains why causal masking allows next-token predictions to be trained in parallel, and why upcasting operations such as softmax and RoPE can prevent numerical problems. He also demonstrates how Unsloth uses Triton kernels and memory offloading to make fine-tuning practical on modest GPUs. The final section covers Gemma fixes, Phi-3's sliding-window and fused-weight issues, tokenizer problems, chat templates, CSV conversion, and exporting fine-tuned models to Ollama. Han is candid that some architecture choices remain empirical rather than well understood.
Han explains that his work began with a Gemma activation-function discrepancy. Different implementations used exact GELU or approximate GELU, and this led to a broader search that found several other issues. He describes similar investigations into Mistral tokenizers, Llama 3 untrained tokens, NVIDIA's NeMo model, and Phi-3. His method is to read the paper, inspect the original implementation, compare framework implementations line by line, and measure differences at each layer. He says this work still has a human guessing step because engineers must decide which implementation matches the model creator's intent. He wants the process to become an open-source effort where people can find and fix bugs in released models.
Han treats tokenization as a separate source of model bugs. He gives Mistral variants as an example, where a smiley and a space can be tokenized differently depending on the model. Some differences came from the Mistral team not updating a model to the fast-tokenization variant. He constructs a deliberately simple tokenizer in the workshop, then discusses problems with punctuation, capitalization, stemming, vocabulary size, and subword methods such as WordPiece and BPE. His broader point is that a model can have a broken tokenizer before its neural architecture is even loaded. Changing a tokenizer after most training is complete can require retraining, so he recommends small experiments before scaling up.
Unsloth trades implementation work for faster fine-tuning
Han describes Unsloth as a way to make fine-tuning easier on limited hardware. He says the package targets models such as Llama, Gemma, and Mistral, uses Triton kernels, and aims for faster training with lower memory use without accuracy loss. He recommends free or inexpensive GPU environments such as Kaggle and Google Colab, while warning that advertised FLOP numbers can include sparsity or use different floating-point formats. He explains that NVIDIA sparsity can skip matrix multiplications involving zero weights, but models generally need to be trained with sparsity for this to be safe. He also discusses gradient-checkpointing and offloading to system RAM, which can increase context capacity with a small execution-time cost when implemented correctly.
Han writes the Llama-style Transformer as a repeated block of operations. Inputs pass through normalization, rotary position embeddings, attention, a residual connection, another normalization, a gated MLP, and another residual connection. The block is repeated for the model's layer count, followed by a final normalization and language-model head. He prefers explaining attention mathematically rather than assigning strong meanings to the words query, key, and value. Starting with an embedding matrix X, the model multiplies X by learned matrices to produce Q, K, and V. The attention calculation then uses QK-transpose, scaling, softmax, and multiplication by V. He says the model's apparent complexity becomes easier to inspect once unnecessary framework code is separated from the core equations.
Han explains that a decoder model predicts the next token from tokens to its left. A sentence such as 'hello my name is Daniel' can be shifted by one position so that the input and target sequences line up, with an end-of-sentence token filling the final gap. The causal attention mask prevents a token from seeing future tokens during training. This lets the model calculate many next-token predictions in one pass instead of separately processing 'hello', then 'hello my', then 'hello my name', which would have much worse scaling. Han repeatedly warns that future-data leakage can produce suspiciously high accuracy in machine-learning papers and competitions. He also stresses careful train-test splitting, shuffling, stratification, and inspection of the data distribution.
Precision choices affect both speed and correctness
Han connects low-precision arithmetic with the implementation bugs he finds. Lower precision can make matrix operations faster, but values have less range and precision, so operations such as softmax and rotary embeddings may need to be upcast to float32. He recommends subtracting the maximum value in each row before exponentiating in softmax to reduce numerical instability. For Gemma, he says the original implementation's upcasting behavior had to be followed rather than copied from Llama or Mistral code. He also explains that advertised speedups depend on the floating-point format and on hardware support, so engineers should distinguish float32, float16, bfloat16, and float8 rather than accepting a headline FLOP number.
Gemma and Phi-3 show why reference matching matters
Han shows a Gemma comparison that measures the layer-by-layer L2 error between the original implementation and other versions. Several fixes reduce the error, with the float32 RoPE correction producing a particularly large change. He says the final choice of fixes was guided by matching the original implementation, even when another combination appeared to produce a lower error in one graph. For Phi-3, he discusses a 2048-token sliding-window setting that appeared as 2047, along with fused Q, K, and V weights that he says can hurt fine-tuning with low-rank adapters. His advice is to inspect configuration files and model code carefully, then test each suspected difference instead of assuming that a popular framework implementation is correct.
Fine-tuning data and chat templates must match the serving format
Han describes a workflow that turns fine-tuning data into the format expected by a chat model and by Ollama. The training example needs an instruction and a response, so extra columns in a CSV must be merged into the instruction or converted into text. He shows customizable chat templates for several formats, including Llama 3 and Gemma, and says the template must be applied consistently during training and model export. His export flow saves adapters, merges weights when needed, quantizes to formats such as GGUF, and creates an Ollama model file from the chat template. He warns that incorrect templates can produce poor responses even when the underlying fine-tuning succeeded. The workflow also supports running inference on examples such as a Fibonacci continuation and serving the result through Ollama.
"You must always follow the original implementation if you don't follow the original implementation then you will get wrong, somewhat worse results."2:00:23
Who should watch
You are implementing or fine-tuning an open-weight model and need to check whether its framework code matches the model's paper or reference implementation.
Your training run has NaNs, suspiciously high accuracy, poor fine-tuning behavior, or unexplained differences between model libraries.
You want to fine-tune on modest GPUs and need practical guidance on precision, memory offloading, chat templates, and model export.
