Embeddings contain more recoverable detail than a row of numbers suggests, including topics, structure, sentiment, and sometimes proper nouns.
2
Vector arithmetic can manipulate text and images by moving through directions associated with properties such as length, sentiment, style, and subject.
3
Making embeddings visible and interactive gives engineers a practical way to study model representations and build new interfaces around them.
Summary
Linus Lee presents embeddings as spaces that can be inspected and manipulated rather than as opaque arrays used only for search. He demonstrates an encoder-decoder system that reconstructs text from embeddings, then moves embeddings along learned directions to change length or sentiment while preserving much of the original topic. Mixing parts of two embeddings produces text with properties from both sources. A linear adapter also lets his decoder recover details from OpenAI's Ada 2 embedding space. In image space, CLIP interpolation creates transitions between photographs and cartoon images, while text-derived vectors alter properties such as sadness or beach content. Lee explains the custom T5-based text model, its decoder, and the adapter used for other embedding models. He points to automated feature discovery work from Anthropic and other labs, then makes his models and a Colab notebook available on Hugging Face. The talk argues for interfaces that let people see and directly handle model representations.
Prompting gives indirect control, so latent spaces may provide a closer control layer
Lee compares prompting a language model to steering a car from the back seat with a pool noodle. Prompts affect the model through several layers of indirection, and he expects that barrier to remain even as models improve at following instructions. Latent spaces offer a possible closer layer of control. Embeddings are high-dimensional vectors, such as 1,536 or 1,024 numbers, and related spaces also appear in model activations, token embeddings, and image models. Lee wants to understand the features represented there and intervene directly in those representations instead of relying only on tokens.
An encoder and decoder can make semantic changes to an embedding visible
Lee shows a custom model that turns a sentence about Greg Egan's novel Diaspora into an embedding with 248 values. A decoder reconstructs the source sentence, including details such as the book title, although the output is not identical. Blurring the embedding and sampling nearby points causes semantic drift: the title changes, while some details such as the author's name remain. More useful directions can change a property while preserving the subject. Moving along a direction associated with shorter text produces increasingly concise versions of the sentence, which can then be arranged in a canvas interface.
Projecting text onto learned directions reveals the attributes an embedding model uses
Lee compares an objective, Wikipedia-style passage with a version rewritten by ChatGPT to sound pessimistic. He projects both embeddings onto directions associated with negative sentiment, artistic content, and length. The rewritten passage lies much farther along the negative sentiment direction, while its artistic topic and approximate length remain similar. This gives a way to ask where text lies along a property represented in the model. The method does not require treating the whole embedding as a single meaning. It exposes separate attributes that can be measured and explored.
Combining parts of embeddings can produce text with properties from both inputs
Lee takes two text embeddings, one describing Diaspora and another beginning a short story about an old Mediterranean coastal town. He constructs a new 2,000-dimensional vector by taking roughly the first half of one embedding and the last half of the other. Decoding that combined vector produces text with features from both sources. The output shares structural details such as a quoted book title, as well as topic, punctuation, and tone similarities. Lee presents this as a simple example of latent-space arithmetic. The result is imperfect, but it shows that the dimensions can interact in meaningful ways.
A decoder can recover surprising detail from another model's embedding space
Lee adapts his decoder to read embeddings from OpenAI's Ada 2 text embedding model rather than his own model. A trained linear adapter maps the external embedding into the decoder's expected space. The reconstruction is less exact, but it retains substantial information without access to the source sentence. Lee points to the proper noun Diaspora, the quoted book-title structure, the topic, and occasional references to the author. He uses these examples to argue that high-capacity embedding spaces encode details such as punctuation, structure, and names, even when those details are not directly visible in the numeric vector.
The same arithmetic works in image space, with uneven results
Lee uses CLIP embeddings to interpolate between a photograph of himself and a cartoon version of himself. The generated images move gradually from photographic features toward cartoon features. He also adds and subtracts text-derived image vectors, such as subtracting a smiling man and adding a sad crying man, then tries to add a beach vector to an image. The sadness manipulation produces a clear result, while the beach feature does not survive that example. Lee stresses that these prototypes mainly calculate embeddings, add or subtract vectors, apply normalization, and generate samples.
Automated methods may discover interpretable features without hand-picked examples
Lee explains that many of his feature directions come from examples and centroids. He then points to recent work from Anthropic, Conjecture, and other labs that searches for dimensions inside models without supervision. He is interested in latent spaces that encode representations of model inputs and outputs which are interpretable and controllable. The talk connects this research to the practical question of making otherwise invisible model activity observable. Lee sees feature discovery as a way to understand what models are using, rather than treating their internal spaces as inaccessible.
Small interactive tools can turn model representations into a thinking medium
Lee describes his text model as a modified T5 encoder-decoder with pooling layers that create an embedding and a gated layer that lets the decoder reconstruct text from it. For a different embedding model, only a linear adapter needs to be trained before using the decoder. He releases the models on Hugging Face along with a Colab notebook for interpolation and feature interpretation. His broader claim is that complex model data becomes easier to understand when it can be seen, handled, and directly manipulated. He connects this idea to Brett Victor's description of a thinking medium as a way to bring thoughts outside the head.
"If you can get something into a thing that feels like it can fit in your hand that you can play with, that you can concretely see and observe and interact with, can be directly manipulated, visualized, all these things, the tools and prototypes that you can build around these things I think help us get a deeper understanding of how these models work."16:58
Who should watch
You are building systems around embeddings and want to inspect what information they retain beyond nearest-neighbor search.
You want practical examples of changing text or image properties through vector operations.
You are exploring model interpretability and need a starting point for interactive tools, adapters, or feature discovery.
