# Reinforcement Learning, Kernels, Reasoning, Quantization & Agents

Daniel Han, Unsloth | AI Engineer World's Fair 2025 | 2:42:28

Source: https://www.youtube.com/watch?v=OkEGJ5G3foU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/reinforcement-learning-kernels-reasoning-quantization-agents
Published: 2025-07-19
Tags: fine-tuning, inference, open-models, reinforcement-learning

## TL;DR
- Reinforcement learning can improve language models when the reward is verifiable, but designing the reward function is harder than choosing the optimization algorithm.
- GRPO removes the value model from PPO and estimates relative performance by sampling several answers for each question.
- Models can become much smaller through dynamic quantization, while selected layers remain at higher precision to preserve accuracy.

## Summary
Daniel Han gives a practical workshop on reinforcement learning for language models, with examples from Unsloth's GRPO notebooks. He explains how models move from pre-training through supervised fine-tuning and post-training, then connects language-model training to the agent loop of states, actions, and rewards. His main argument is that reward functions matter more than the specific RL algorithm. Mathematical correctness can be checked directly, while code and open-ended tasks need more involved checks, such as execution or an LLM judge. He breaks down PPO and GRPO, including the purpose of the advantage, likelihood ratio, clipping, and KL divergence. A live notebook shows how a base model can learn reasoning traces through supervised priming and GRPO. The final section covers dynamic quantization, super weights, low-bit formats, and kernel optimization with torch.compile.

## Key ideas
### Language models improved in stages, and reasoning created a new performance jump
[19:00](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=1140s)
Han describes pre-training as the stage where a model learns from broad data, followed by supervised fine-tuning that turns a base model into a chat model. Preference fine-tuning and reinforcement learning come later. He compares the stages to Yann LeCun's cake analogy: pre-training is the cake, supervised fine-tuning is the icing, and reinforcement learning is the cherry. He calls supervised fine-tuning and human feedback the first major jump, followed by an RL jump. DeepSeek-R1 showed that an open model could reach the level of reasoning models such as OpenAI's o1 and o3. Han thinks many reasoning abilities may already exist in the model and that RL makes them appear more often.

### An agent loop turns model outputs into actions that receive rewards
[16:56](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=1016s)
Han defines an agent as something that acts in an environment and receives a reward after taking an action. In a game such as Pac-Man, the state changes after every move, so the agent needs a history of what happened. Language-model RL is simpler in some ways because many prompts can be treated as separate single-turn tasks. For 'What is 2 plus 2?', the prompt is the state, the generated answer is the action, and a function assigns a reward. The reward can be positive or negative, and it can use exact correctness or distance from the correct answer. Han says OpenAI's reasoning systems can be understood as using collections of reward functions across tasks.

### Reward functions are harder to design than the RL algorithm
[30:48](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=1848s)
For mathematics, a reward function can check whether the answer is correct, give partial credit when a number is close, or assign a larger reward to an exact answer. For code, the function can check whether the program runs, imports the expected library, or produces a requested output. Open-ended tasks such as creating a Flappy Bird game are harder because successful execution does not fully prove that the result is good. Han suggests combining checks, including an LLM judge, human review, or visible properties of the output. He is direct that reward design is the difficult part. The algorithm can be swapped more easily, while a bad reward can teach the model to exploit the scoring rule.

### PPO adds safeguards around the basic policy-gradient update
[51:22](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=3082s)
Han starts with the REINFORCE idea of increasing the probability of actions with good outcomes and decreasing the probability of bad actions. He then explains why PPO adds more terms. The advantage compares an action's reward with an estimated average reward, so the model learns whether an action is better or worse than its current baseline. PPO uses the likelihood ratio between the updated policy and the policy that produced the action. Clipping limits how far the update can move in one step. A KL-divergence term penalizes the model for moving too far from the supervised fine-tuned model. Han describes these additions as ways to reduce overfitting and keep training stable.

### GRPO estimates relative quality by sampling several answers
[1:16:30](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=4590s)
GRPO, the method Han associates with DeepSeek-R1, removes the value model used by PPO. Instead of training another large model to estimate the average reward, it samples several answers for the same question. The rewards from that group are normalized using their mean and standard deviation, producing a relative score. For 'What is 2 plus 2?', four rollouts might produce zero, one, two, and four, with only four receiving the positive reward. The correct answer then has the highest relative score, while the other samples are pushed down. Han says the group-relative calculation saves parameters and compute. He also emphasizes that sampling needs variability, so temperature should not be zero.

### Supervised priming helps RL escape a zero-reward starting point
[2:24:45](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=8685s)
Han shows why starting GRPO directly from a base model can be inefficient. A base model may not know how to follow an instruction or put its answer in a usable format, so its reward can remain zero across many samples. The notebook first uses a small supervised fine-tuning set to teach the model a reasoning format. The model is prompted to place its working between markers and its answer between solution markers. DeepSeek-R1 outputs provide examples for this priming stage. Han says the demonstration used only a small number of rows, then GRPO searches for occasional good answers and increases their probability. The training run took about two hours and 54 minutes on a free Colab GPU, with rewards improving over time.

### Dynamic quantization keeps sensitive layers at higher precision
[2:33:17](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=9197s)
Han explains that a large DeepSeek-R1 model can be reduced substantially with low-bit quantization, though some accuracy is lost. The important choice is which layers to quantize. In mixture-of-experts models, he recommends quantizing expert layers more heavily while leaving attention, shared-expert, and other sensitive layers at higher precision. A vision example shows that naively quantizing every layer can make the model describe a train image as a coastal scene. Leaving selected layers unquantized restores the answer. He points to the Super Weights paper, which argues that a small set of weights can matter greatly even when they are not numerical outliers. Activation and weight quantization errors can help locate these sensitive parts.

### Lower numerical precision and compiled kernels drive much of the speedup
[2:35:58](https://www.youtube.com/watch?v=OkEGJ5G3foU&t=9358s)
Han attributes much of the improvement in GPU speed to lower numerical precision, moving from float32 to float16, bfloat16, float8, and eventually float4. He expects float4 to be close to the practical limit for this source of speedup because there are few useful bits left after that. He also recommends using torch.compile, while warning that it has many options and does not improve every workload automatically. Unsloth's optimizations include Triton kernels, shared weights between inference and training, gradient checkpointing, and parameter-efficient fine-tuning with LoRA. The notebook is designed to reduce memory use enough for experiments on free Colab or Kaggle GPUs.

## Notable quotes
- "The hard part is actually the reward function itself and the data that you're going to shove into the model." (1:30:49)
- "Everything's reducing overfitting. Right? That's all of machine learning and AI. It's just to make the training more stable and to reduce overfitting." (1:59:44)
- "The goal of GPO is suddenly we see a good answer with a good reward we want to maximize that and that's the whole point of GPU." (2:19:47)
- "You must use torch.compile. You know, every single function that you see, wrap it in torch.compile." (2:41:03)

## Tools & references mentioned
- Unsloth
- Hugging Face
- Google
- Meta
- Mistral
- Llama
- Gemma
- Phi
- Qwen
- DeepSeek-R1
- OpenAI o1
- OpenAI o3
- Yann LeCun
- Maxim
- PPO
- GRPO
- REINFORCE
- RLVR
- RLHF
- DPO
- LoRA
- Triton
- vLLM
- SGLang
- torch.compile
- Super Weights paper
- Nathan Lambert's policy gradients book

## Who should watch
- You want to fine-tune an open language model for reasoning, tool use, mathematics, or code and need a concrete starting point for reward functions.
- You are comparing PPO and GRPO and want an explanation of why GRPO removes the value model.
- You need to run fine-tuning or inference on limited GPU memory and want practical advice on priming, LoRA, quantization, and compiled kernels.

## Related talks

- [Special Topics in Kernels, RL, Reward Hacking in Agents](https://aietalks.com/talks/special-topics-in-kernels-rl-reward-hacking-in-agents) (Daniel Han, Unsloth, 2:20:21)
- [Reasoning + RL](https://aietalks.com/talks/reasoning-rl) (Will Brown & Greg Kamradt, ARC Prize Foundation & Aakanksha Chowdhery, Reflection AI & Ryan Marten, Bespoke Labs & Kyle Corbitt, OpenPipe & Nathan Lambert, AI2 & Christian Szegedy, Former co-founder of xAI, 3:54:58)
- [Reinforcement Learning without Verifiable Rewards](https://aietalks.com/talks/reinforcement-learning-without-verifiable-rewards) (Will Brown, Prime Intellect, 19:27)
- [Training Agentic Reasoners](https://aietalks.com/talks/training-agentic-reasoners) (Will Brown, Prime Intellect, 19:17)
- [Open Questions for AI Engineering](https://aietalks.com/talks/open-questions-for-ai-engineering) (Simon Willison, Independent open source developer, 24:33)