A working GPT-2 Small implementation in Excel can make tokenization, embeddings, attention, neural networks, and prediction concrete for engineers without machine learning training.
2
GPT-2 uses residual connections as an information stream, with attention moving information between token positions and perceptrons processing each position.
3
Sparse autoencoder features can be injected into the residual stream to steer a model, such as making GPT-2 turn Mike's phone into a lightsaber.
Summary
Ishan Anand gives a spreadsheet-based tour of GPT-2 Small, implemented entirely in Excel with no Python or API calls. He starts with the path from text to subword tokens, embeddings, transformer layers, and a probability distribution over the next token. He then opens the attention heads, showing how tokens can use earlier context, and explains the multi-layer perceptron as matrix multiplication followed by an activation function. Residual connections let information move through the model while layers read and write to a shared stream. Using a logit lens, he examines predictions after each layer and describes a circuit that helps GPT-2 answer day-of-the-week prompts. The final section demonstrates model steering by adding a sparse autoencoder feature for Jedi into the residual stream, causing a prompt about Mike to produce a lightsaber. Anand connects model architecture to practical prompting, including RWKV's different context limitations.
A spreadsheet can expose GPT-2's full numerical pipeline
Anand presents GPT-2 Small as a patient on an operating table made from an Excel spreadsheet. The workbook implements the model in pure Excel functions, with no API calls and no Python. It has more than 150 tabs and over 124 million cells, including the model's parameters. He says a reader could understand GPT-2 by moving through the workbook tab by tab and function by function, although the talk gives an abbreviated tour. The model takes input text, converts it to tokens, maps those tokens to numbers, runs the transformer calculations, and converts the result back into a predicted next token.
Tokenization turns phrases into subword units before the model does any reasoning
The model cannot work directly with a sentence such as 'Mike is quick he moves.' Anand explains that it first splits text into tokens and maps those tokens to embeddings. A word may be one token, but it may also be split into several subword units. The spreadsheet shows 'funology' being divided into 'fun' and 'ology', while 'reinjury' is split into 'rain' and 'jury'. This comes from an algorithm that chooses common subword units, which can produce results that do not match a person's intuition. In GPT-2 Small, each token embedding contains 768 numbers.
Attention uses earlier tokens to resolve the meaning of ambiguous words
The transformer has 12 blocks, each containing attention and a multi-layer perceptron. Anand describes attention as allowing tokens to look at surrounding context, while a causal mask prevents them from looking forward. In one attention head, the token 'Mike' attends to itself, while 'he' assigns about 0.48 attention to its antecedent. The word 'quick' can refer to movement, intelligence, a part of a fingernail, or something alive in Shakespearean English. Seeing 'moves' helps the next computation favor a meaning related to physical movement, leading toward words such as 'quickly' or 'fast'.
Residual connections let transformer layers pass information around each other
Anand explains residual connections as addition operations inside every layer. They allow information to route around or skip parts of the attention and perceptron calculations. He describes the residual stream as a communication network or information highway, with one lane for each token position. Attention moves information across lanes, while the perceptron processes information within each lane. Because the layers read and write to this shared stream, later layers can use information created earlier and can alter predictions without every component needing to process the same information in the same way.
A logit lens shows when a prediction appears and changes during the forward pass
The logit lens applies the language head between every layer to inspect what the model would predict at that point. For the prompt 'If today is Tuesday tomorrow is', GPT-2 eventually predicts 'Wednesday'. The answer appears near the start of the process, disappears, and returns near the final layers. At one point, tokens related to time and days become more likely, while 'Wednesday' remains only the third-ranked option. Anand describes research that isolated four components involved in solving this day-of-the-week task: a perceptron in layer zero, attention from layer nine, a single attention head, and a perceptron in layer nine followed by attention from layer ten.
Sparse autoencoder features provide a way to alter the residual stream
Anand describes work on sparse autoencoders that separates the residual stream into interpretable features. Researchers can identify a feature, then increase or decrease its contribution in the stream. He shows a GPT-2 Small feature associated with Jedi and adds its decoder vector to the start of a block. The change is simple spreadsheet arithmetic: multiply the feature vector by a coefficient and add it to the residual stream. Other steering approaches he names include representation engineering, which can derive a vector with principal component analysis, and activation steering, which can subtract a phone-related vector from a Jedi-related vector before injecting the result.
Model architecture affects how prompts should be written
Anand argues that understanding architecture helps engineers reason about a model's behavior, limits, and prompting rules. He compares a normal Transformer prompt template with the template recommended for RWKV. RWKV has a different type of model mechanism, and its pseudo-attention cannot look backward in the same way as regular Transformer attention. Because of that limitation, its instructions and context use a different order. The example connects an internal architectural property to an external engineering decision: the prompt format itself can need to change when the model architecture changes.
"The residual stream here is every one of those tokens and information is flowing through them like an information Super Highway."09:41
Who should watch
You work with language models but want a concrete explanation of tokenization, attention, residual streams, and prediction without starting from code or advanced mathematics.
You need to explain model behavior to non-ML engineers or stakeholders and want an inspectable example rather than a collection of terms.
You are exploring interpretability or activation steering and want to see how a feature can be added directly to a model's residual stream.
