Text diffusion generates a block of tokens through repeated denoising steps, which can reduce memory transfers compared with generating one token at a time.
2
Bidirectional attention lets a diffusion model use future tokens to revise earlier mistakes and spend more denoising steps on difficult prompts.
3
Text diffusion has lower latency but lower throughput at large batch sizes, so it is currently expensive for large-scale serving.
Summary
Brendan O'Donoghue explains how text diffusion applies the denoising process used for images to sequences of tokens. A model starts with corrupted or random tokens, then refines the whole sequence over several forward passes. Compared with autoregressive generation, this can make better use of GPU and TPU hardware. For example, 24 denoising passes can generate 256 tokens while requiring about 10 times fewer memory transfers than 256 individual token-generation steps. Diffusion also has access to future tokens, which lets it revise an answer after completing its reasoning. The model can vary its compute budget by stopping quickly on easy prompts and taking longer on difficult ones. It can edit text in place, such as fixing code or inserting a paragraph. The main limitation is serving cost: autoregressive models achieve better throughput with large batches, while diffusion reaches its compute limit sooner. The demos show low latency enabling generated websites, operating systems, and voice-built software.
Text diffusion learns to restore corrupted token sequences
The training process starts with a clean sequence, such as a sentence, and gradually corrupts it by adding or replacing tokens. The model learns to fill in the missing or incorrect information at different noise levels. During inference, the sequence begins as random discrete tokens. The model then refines the whole canvas over several denoising steps until it produces a relatively clean response. Brendan O'Donoghue describes the process as visually similar to image diffusion, where a noisy starting point gradually becomes a finished output.
Diffusion generates blocks over several passes instead of one token at a time
Autoregressive models generate a response one token after another, conditioning each new token on the previous context. Text diffusion initializes a longer sequence, potentially hundreds or thousands of tokens, with noise and refines that sequence over multiple passes. It is therefore not a single forward pass, but each pass handles the whole block together. The model can also attend to future tokens within that block, giving it a different way to construct a response.
Text diffusion is faster because it transfers model data fewer times
Brendan explains that modern GPUs and TPUs often have ample matrix-multiply capacity but limited memory bandwidth. Autoregressive generation is memory-bound because generating each token requires streaming the model weights, activations, and KV cache again. If diffusion generates 256 tokens in 24 passes, it transfers the model data roughly 10 times fewer than an autoregressive system that performs 256 token steps. The resulting speedup depends on the workload being memory-bound. The Gemini Diffusion demo reached about 2,000 raw tokens per second in the example he gives.
Future-token access lets the model correct an earlier answer
In a math example, Gemini Diffusion first writes 60, then changes the answer to 49 after another forward pass. Once it completes the reasoning, it returns to the beginning and changes the answer to 39. Brendan contrasts this with GPT-4o, which answered 40 before later acknowledging the mistake, and Gemini 2.5 Flash, which answered 42 and kept that answer through its reasoning. The diffusion model can revise the earlier token because it sees the future part of its own response.
The model can spend different amounts of compute on different prompts
More denoising steps generally improve output quality, although Brendan says the improvement is not perfectly monotonic. The model can also be trained to decide when it is finished. In his examples, producing the first 100 digits of pi takes four steps, generating a FizzBuzz program takes 18 steps, and explaining quantum mechanics in one paragraph takes 31 steps. He says harder internal evaluations, such as GPQA Diamond, take longer than simpler Python programming tasks such as MBPP.
The model can preserve surrounding context while changing a selected part of a response. In the demos, it fixes an indexing bug in code, adds documentation, and inserts a middle paragraph into a story. This works because the model sees the text on both sides of the edit while filling in the missing section. Brendan connects this behavior to image diffusion, where the model can fill a removed region using the rest of the image and the prompt.
Very low latency enables generated interfaces to respond like software
The demos use low latency to generate whole interactive experiences on demand. One system creates a Wikipedia-like page, including its HTML and text, when the user clicks a link. Another creates a Reddit-like site with bot comments, generated images, and generated page structure. A third generates an operating system response for every click, including its pages and text. In a separate demo, voice instructions create a todo app, add random tasks, mark some complete, add sorting, and switch to dark mode in about 15 seconds.
Serving cost is the main barrier at large batch sizes
Text diffusion has lower latency for an individual request but lower throughput when many requests are batched. Autoregressive models can process a large batch efficiently even though each user waits longer, which helps keep serving costs down. Diffusion performs multiple forward passes over the same data and reaches its compute limit sooner. Brendan says this higher serving cost is why large models have not generally adopted text diffusion, despite its latency advantage. He points to on-device settings such as phones and robots as cases where low latency may matter more than batching many users.
"It had a mistake twice, 60 and then 49, but once it finished the reasoning, it was able to return back and fix the mistake that it made at the start."10:17
Who should watch
You are evaluating text diffusion for a product where response latency matters more than serving thousands of batched requests.
You are building on-device, robotics, or interactive software and want to understand what lower token latency could enable.
You are comparing diffusion and autoregressive generation and need the hardware and serving-cost tradeoffs explained.
