A small language model can be trained from scratch on a laptop with 16 GB of RAM, using PyTorch and a simple character-level tokenizer.
2
The workshop model learns by predicting the next Shakespeare character, with validation loss used to detect overfitting and monitor progress.
3
Base models and reasoning models often share the same architecture, while high-quality post-training data teaches the reasoning behavior.
Summary
Angelos Perivolaropoulos walks through the parts needed to train a small GPT-style language model without pretrained weights. The workshop uses PyTorch, a Shakespeare dataset, and a character-level tokenizer with 65 tokens. He explains how token embeddings, positional embeddings, attention, MLP layers, residual connections, and layer normalization fit together in a GPT-2-style model. The training loop uses next-token prediction, cross-entropy loss, AdamW, learning-rate warm-up, cosine decay, checkpoints, and validation loss. He also covers inference with temperature and top-k sampling, then explains how to organize the project into model.py, train.py, and generate.py. The practical advice is direct: start small, inspect the loss curves, and treat rising validation loss as a sign of overfitting. In the Q&A, he describes reasoning as a post-training process driven by high-quality chain-of-thought data, and connects text-model training to audio and multimodal systems through encoders and shared embedding spaces.
The workshop reaches most of the way to a real model with simple tools
Angelos Perivolaropoulos says the workshop trains an LLM from scratch, with no pretrained weights and no Transformers library. The code uses PyTorch and basic libraries, which he considers a useful level for understanding how research engineers design models. He estimates that the workshop covers about 80% of the process of creating a model from scratch. The remaining work involves optimizations, scaling, and adapting the model to particular use cases. Participants can run the project on a laptop with 16 GB of RAM, use Apple Silicon, CUDA, or a CPU, or switch to Google Colab for a free GPU.
The tokenizer controls whether a small dataset can support training
The example uses character-level tokenization because the Shakespeare dataset contains only 65 distinct characters. That produces 4,225 possible bigrams, since each character can be followed by another character. Angelos says this is manageable for the available data, while a 200,000-token vocabulary would require data on a much larger scale to expose enough token combinations. Character tokenization is simple, but it does not scale well because the model has to learn relationships across individual letters. For larger systems, he describes byte pair encoding, which combines common character patterns into reusable tokens.
A GPT-style transformer combines attention with several small components
The transformer in the workshop uses multi-head self-attention, an MLP, residual connections, and layer normalization. Attention lets each token weigh earlier tokens and learn relationships such as the connection between 'sky' and 'blue'. The MLP combines those relationships into a representation from which the model can produce logits. Residual connections pass the previous activation forward instead of forcing every layer to rebuild it. Layer normalization keeps activations from growing uncontrollably. Angelos presents these as reusable building blocks whose implementations are only a few lines of mathematical code.
The demonstration model is deliberately small and has a 256-token context
The model configuration uses a vocabulary size of 65, a block size of 256, six transformer layers, six attention heads, and an embedding dimension of 384. The block size is the maximum sequence length the model sees during training. Angelos describes 256 tokens as tiny compared with larger systems, where 16,000 tokens can be a common middle ground and some labs target much longer contexts. Increasing context length is not just a matter of changing one number. The architecture and training process must keep attention efficient and training stable.
Next-token prediction is implemented by shifting the target sequence by one position
The training objective is to produce Shakespeare-like verses. The model receives tokens T0 through TN and learns to predict T1 through TN+1. Cross-entropy compares the predicted distribution with this shifted target sequence. The dataset contains about 1 million characters, which the code splits into training and validation portions. Batches contain 64 sequences, each with 256 tokens. The data loader is intentionally simple, and Angelos points out that larger systems need more involved loading strategies, especially when context windows become longer.
Learning-rate scheduling and validation determine whether training is working
Angelos recommends starting with a low learning rate, increasing it during a warm-up, then reducing it with cosine decay. A high rate can move the weights too far and make training unstable. The example uses 100 warm-up steps and trains for 5,000 steps. Training loss should fall, while validation loss measures performance on data the model has not seen. If training loss falls but validation loss rises, the model is overfitting. Loss spikes suggest a bug in the data or training code. If both losses plateau, the current dataset may be exhausted.
Sampling settings control whether generated text is repetitive or varied
For inference, greedy decoding always selects the most likely next token. Angelos says this can work well for transcription, where creativity is unwanted, but often makes language-model output boring. Temperature allows lower-probability tokens to be selected, which can make generation more varied. He gives 0.7 as a useful middle ground. Top-k sampling limits the model to a set of likely tokens, preventing temperature from selecting an extremely unlikely continuation. A fixed seed makes the generation reproducible for a given prompt and checkpoint.
Reasoning behavior usually comes from post-training data rather than a new base architecture
Angelos explains that base, instruct, and reasoning models often share the same underlying model. A base model can be post-trained with high-quality reasoning data, including carefully written chains of thought. He describes this data as expensive because it may come from specialists and must pass quality checks. The model learns to generate reasoning tokens, then attend back to them before producing an answer. He says even older small models can be adapted into reasoning models when the model has enough capacity and the training data is strong.
Audio and multimodal systems reuse the transformer through encoders and embeddings
For audio, the general training ideas remain similar, but the tokenizer and loss change. Audio is often converted into mel spectrograms, and losses such as L2 can compare predicted and target spectrograms. In multimodal models, a separate encoder can process video or audio and produce vectors with the same dimension expected by the language model's embedding input. Those vectors are inserted alongside text representations. The transformer works with the vectors rather than needing to know whether they came from words, sound, or images. Angelos says the hard part is designing representations that work across modalities.
"The base building blocks are very similar. You can train the same exact model, you can post train it, which is how usually reasoning is being taught to this model."1:05:27
Who should watch
You want to understand the full training path from text data to generated output without starting with a large framework.
You have a laptop or Google Colab and want a small project that demonstrates tokenization, transformer code, optimization, and inference.
You work with audio or multimodal models and want a clear explanation of how encoders, embeddings, and task-specific losses connect to language models.
