You can understand a transformer's main operations with JavaScript, basic matrix multiplication, and intuition instead of advanced machine learning coursework.
2
GPT-2 turns text into subword tokens, maps them into learned embeddings, lets tokens exchange context through attention, and repeatedly refines predictions through transformer blocks.
3
ChatGPT differs from GPT-2 mainly through scale and training that adds instruction examples, preference modeling, and reinforcement learning from human feedback.
Summary
Ishan Anand builds a working mental model of GPT-2 by walking through an implementation that runs locally in a browser using vanilla JavaScript. He starts with the model's basic task, predicting the next token, then follows the data through byte-pair tokenization, token and position embeddings, attention, the multi-layer perceptron, repeated transformer blocks, and the language head. The examples are concrete, including subword splits, vector arithmetic, attention percentages, ReLU approximations, logits, softmax, and greedy or top-k sampling. Anand also explains how training adjusts model parameters with backpropagation and gradient descent. The final comparison with ChatGPT separates the base model's internet-text prediction from later instruction tuning and preference training. The workshop is aimed at web developers who want to inspect model behavior without switching to Python or learning advanced calculus first.
A browser-based JavaScript implementation can make GPT-2 inspectable to web developers
Anand's workshop uses a complete GPT-2 implementation in vanilla JavaScript, because many AI engineers come from web development and may continue using TypeScript and Next.js around a model. The model weights load locally into the browser, using about 1.5 GB of disk space, so the implementation can run without an internet connection. Its notebook-like cells contain JavaScript or spreadsheet-style tables that show intermediate results. Developers can use console.log and the browser's debugger to step through the model line by line. Anand says the workshop focuses on why the code works, using analogies and examples instead of relying on complex equations.
An autoregressive language model generates passages by repeatedly predicting one next token
Anand describes a language model's core job as predicting the next token from a passage. Given "Mike is quick, he moves," the model might produce "quickly." To generate more text, the program appends that output to the original input and runs the model again. The next prediction might be "and," followed by another iteration. This repeated input-output loop is why the model can produce paragraphs even though each individual model run predicts only one token. Anand explains that researchers turn the word problem into a math problem by mapping text to numbers, performing many arithmetic operations, then translating the result back into token scores.
Subword tokenization balances vocabulary size with sequence length
GPT-2 uses byte-pair encoding to split text into subword units. Anand demonstrates that "reinjury" can become several tokens, while "reindeer" shares an initial piece but is split differently. Word tokenization would struggle with misspellings, unexpected languages, and new expressions, while a dictionary containing every word would make the vocabulary larger. Character tokenization would keep the vocabulary small but make every sequence and matrix much longer, while individual characters carry less meaning. BPE learns frequent adjacent pairs from a text corpus, adds those pairs to its vocabulary, and repeatedly recompresses the corpus. The resulting pieces sometimes resemble morphemes, but the tokenization process itself has no semantic understanding.
Embeddings give tokens learned positions in a high-dimensional space
A token ID identifies a dictionary entry, while an embedding gives that token a vector of values that can capture useful relationships. In GPT-2, each token has 768 embedding values. Anand compares token IDs with house addresses: an address tells you where to find a house, while its other attributes describe what is inside. A simplified two-dimensional example shows how king minus man plus woman can produce queen. Actual dimensions are not labeled with concepts such as authority or gender, so individual values are not interpretable. Their geometry can still support similarity and relationships. Anand describes embeddings as a compressed co-occurrence matrix, learned during training from the statistical company words keep.
Position embeddings add word order to token meaning
Token embeddings alone do not preserve the order of a sentence. "The dog chases the cat" differs from "The cat chases the dog," so GPT-2 adds a position vector to each token embedding. The original Transformer used sine and cosine functions for these offsets. GPT-2 instead learned its position embeddings during training. Its position matrix has 1024 rows, matching the model's maximum context length, and each row has the embedding dimension. The implementation adds the appropriate position row element by element to each token's 768-value vector. Anand also points out that many modern models replace GPT-2's position method with rotary positional embeddings, or RoPE.
Attention lets tokens share context and resolve meaning
Inside each transformer block, attention allows tokens to exchange information. Anand uses the pronoun "he" finding its antecedent "Mike" as an example. The word "quick" can mean speed, intelligence, a body part, or life in older English, so nearby words help select the relevant sense. In his gravity analogy, relevance acts like distance and the value acts like mass, allowing tokens to push and pull each other in embedding space. The attention matrix is causal: its upper triangle is zero because a token cannot look at future tokens. Each row sums to one, so its values show how attention is distributed. Multiple attention heads provide separate interaction patterns.
The MLP learns nonlinear mappings through matrix operations and backpropagation
The multi-layer perceptron is the second major operation in each block. A neuron multiplies inputs by weights, adds a bias, and applies an activation such as ReLU, which outputs zero for negative values and passes positive values through. A network of these neurons can be written as matrix multiplication. Anand uses a parabola to show how combining ReLU-shaped pieces can approximate a function, with more neurons producing a closer fit. GPT-2's MLP takes 768 input values, expands to a hidden layer four times wider, applies GELU, and projects back to 768 values. Backpropagation compares the prediction with the known next token and adjusts parameters, including embeddings and attention weights, through gradient descent.
Repeated blocks refine the predicted embedding before token selection
GPT-2 small repeats the attention and MLP operations across 12 blocks. Each block performs the same kind of work, but each has different learned parameters. Anand's implementation changes the block index in its formulas and reruns the operations to mirror this repetition. After the final block, layer normalization produces a predicted next-token embedding. The language head compares that vector with every row in the token-embedding matrix. GPT-2's matrix has 50,257 token rows and 768 columns, producing one score, or logit, for each vocabulary token. Softmax converts those scores into probabilities. Greedy sampling chooses the highest-scoring token, while top-k and top-p sampling restrict the random choice to likely candidates.
ChatGPT adds behavior training to GPT-2's next-token objective
GPT-2 is trained to continue internet text, so it can imitate forms, code, or useful factual passages without being a helpful assistant. Anand shows that a prompt beginning with "First name" can lead to more form fields, and "Hello class" can lead into Java code. Later systems keep much of the same Transformer architecture while changing scale and some components, then add training stages. InstructGPT and ChatGPT use examples of desired assistant responses, followed by preference data containing chosen and rejected answers. A scoring model learns which response people prefer, and reinforcement learning trains the language model toward higher scores. Anand describes this as moving from imitation of text toward optimization of preferred behavior.
"You don't need all that sophistication if you just want to have a really accurate model of how a transformer works."02:29
Who should watch
You build applications with JavaScript or TypeScript and want to understand what happens inside the language model behind an API.
You have no formal machine learning background but can follow basic matrix multiplication and want a runnable GPT-2 implementation to inspect.
You need to distinguish base-model text completion from instruction tuning, preference modeling, and reinforcement learning.
