High-quality generative models depend heavily on carefully curated data, and improving the data can be a better use of time than tuning the model.
2
Modern image and video diffusion models usually operate on learned latent representations because full pixel tensors become too large at higher resolutions and longer durations.
3
Guidance improves prompt alignment and sample quality by trading away diversity, while distillation reduces the number of denoising steps needed for generation.
Summary
Sander Dieleman gives a practical overview of the components behind large-scale image and video generation models. He argues that data curation deserves far more attention than it usually receives, then explains why models generally work in learned latent spaces rather than directly on pixels. The talk focuses on diffusion: gradually corrupting data with noise, learning to denoise it, and sampling through repeated small updates. Dieleman describes diffusion as a form of spectral autoregression that generates coarse structure before fine detail. He compares U-Nets with transformers, discusses spatial and temporal modeling choices for video, and briefly covers sharding and parallelism with JAX. Sampling receives more attention, especially guidance, which improves quality by reducing diversity. He then explains consistency models and related distillation methods for reducing sampling steps. The final section covers text, reference, camera, and timing controls, plus post-training methods for adding conditioning and preferences.
Dieleman says data curation is essential for high-quality results, yet research practice often gives it too little attention. During his PhD, researchers were encouraged to use predefined datasets so results could be compared with existing work. Large generative models require a different habit: teams need to inspect and improve the data itself. He says time spent improving data can sometimes be a better investment than tweaking the model or its optimization. He also explains why he cannot share many concrete details. Data choices are part of what makes these systems good, and companies treat them as confidential.
Learned latents make large visual examples fit in memory
A 1080p video lasting 30 seconds at 30 frames per second can require several gigabytes just to store one training example as pixels. Dieleman explains that current systems therefore learn their own compressed representation with an autoencoder. The encoder maps an image or video through a bottleneck into latents, a generative model operates on those latents, and the decoder converts generated latents back to pixels. Unlike JPEG or video codecs, these representations preserve the grid structure that visual neural networks use. They compress local texture and fine detail while retaining much of the image's semantic and spatial structure.
Diffusion generates by repeatedly reversing a noise process
Diffusion starts with a corruption process that adds increasing amounts of Gaussian noise to an image or video. A denoiser learns to infer a clean version from a noisy input. A one-step prediction is blurry because many clean images could have produced the same noisy observation, so the model effectively predicts a region of possible image space. Sampling takes a small step in the predicted direction, then asks the denoiser again. Some samplers add a little fresh noise after each step, which can reduce the effect of feeding the model its own errors. The process repeats until the noise is gone.
Diffusion follows visual structure from coarse frequencies to fine detail
Dieleman uses Fourier analysis to explain why diffusion fits image and video generation. Natural images have a power-law frequency spectrum, while Gaussian noise contains frequencies in roughly equal measure. As noise is added, it first obscures the highest-frequency details, then progressively hides lower-frequency structure. This means the corruption process removes fine detail before global form. Dieleman describes diffusion as 'spectral autoregression': sampling starts with coarse features and gradually adds finer ones. He says this gives the model a natural way to sketch the semantics of an image before filling in detail, and lets training place more weight on perceptually important scales.
Transformers can replace U-Nets, while video allows hybrid designs
Early diffusion denoisers used U-Nets, convolutional networks designed for tasks where the output has the same dimensions as the input. Dieleman says transformers also work for denoising, although they do not use a causal mask because attention can be fully bidirectional. Their use also allows teams to reuse scaling knowledge developed for language models. For video, systems can jointly noise and denoise the full spatial and temporal volume. Another option is a hybrid: generate time autoregressively, frame by frame, while using diffusion to generate each frame. Dieleman gives Genie as an example of this compromise.
Large-scale training needs data and model parallelism
Visual diffusion models are scaled across many chips, although Dieleman says they are generally still smaller than large language models. At first, data parallelism can split a batch across devices. Eventually, the model itself must be sharded across chips. Dieleman says the Google DeepMind teams use JAX, whose tooling can distribute models and reduce communication between chips. He attributes JAX's design partly to Google's long use of TPUs, where taking advantage of the fast interconnect usually means working across multiple chips. He also makes clear that PyTorch can support these forms of parallelism, so the approach is not unique to JAX.
Guidance improves quality by amplifying conditional information
Guidance changes sampling by making two predictions at each step, one without the conditioning signal and one with it, such as a text prompt. The difference between those predictions is amplified and used to define the denoising direction. This raises sample quality and prompt alignment while reducing diversity. Dieleman says guidance is now routinely enabled because samples can look much worse without it. A guidance scale that is too high can produce oversaturated images. He also describes work that varies the scale during sampling, with less guidance at the beginning and end and more in the middle.
Distillation reduces sampling steps instead of shrinking the model
In diffusion, distillation usually targets generation speed rather than model size. A sampler traces a nonlinear path from noise to a data point, and the diffusion model predicts a local tangent to that path at each step. Consistency models try to predict the eventual endpoint directly, which could allow one-step sampling. Dieleman says this is difficult because one pass must reproduce what the original model learned over many passes. A practical compromise applies consistency modeling over selected intervals, allowing a small number of steps instead of one. He also notes that deterministic sampling is useful for many distillation methods.
Text prompts provide a common way to control image and video models, but Dieleman says users increasingly need reference-based and structured controls. A photo or short video could condition a model so that a generated video includes the person shown, while camera movement, event speed, and event timing may need explicit signals. Since such information is often absent from the pre-training data, it can be added during post-training. Dieleman mentions several ways to inject conditioning into a transformer, including extra tokens or broadcasting information to all tokens. Post-training can also use human preferences through reinforcement learning or direct preference optimization.
"It turns out this is super powerful. It seems almost childish, like it seems so simple, but actually there's sort of like a Bayesian reasoning type thing behind this that makes this work really well."26:07
Who should watch
You are building an image or video generator and need a compact overview of the data, representation, diffusion, architecture, and sampling choices involved.
You are deciding whether to work in pixel space or latent space, or whether a U-Net, transformer, or hybrid temporal design fits your visual model.
You need to improve generation speed or control and want to understand guidance, consistency models, distillation, and post-training conditioning.
