Diffusion Models

What is a Diffusion Model?

Popularized by DDPM (Sohl-Dickstein et al., 2015; Ho et al., 2020) and Stable Diffusion, Diffusion Models represent the state of the art in generative image, audio, and video synthesis.

Instead of competing against a Discriminator like GANs, Diffusion Models learn to generate data by reversing a gradual noise process:

  FORWARD NOISING PROCESS (q - Fixed, No Learning):
  Real Image x_0 ──► Add Gaussian Noise ──► x_1 ──► ... ──► Pure White Noise x_T  (t = 1000 Steps)

  REVERSE DENOISING PROCESS (p_theta - Trained Neural Net):
  Pure White Noise x_T ──► [ U-NET / DiT ] Denoiser ──► x_t-1 ──► ... ──► Generated Image x_0!

The Forward & Reverse Mathematics

1. Forward Process ($q$)

Adds Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$ incrementally across $T$ steps according to a noise schedule $\beta_1, \dots, \beta_T$:

$$q(x_t \mid x_0) = \mathcal{N}\left(x_t; ; \sqrt{\bar{\alpha}_t} x_0, ; (1 - \bar{\alpha}_t) I\right)$$

Using the reparameterization trick, we can jump directly to any noisy state $x_t$ at arbitrary time $t$ in a single closed-form step:

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$$

2. Reverse Process ($p_\theta$) & Training Objective

A neural network $\epsilon_\theta(x_t, t)$ is trained to predict the exact noise vector $\epsilon$ that was added to $x_0$ to produce $x_t$:

$$\mathcal{L}{\text{simple}}(\theta) = \mathbb{E}{t, x_0, \epsilon} \left[ \left| \epsilon - \epsilon_\theta(x_t, t) \right|^2 \right]$$

The loss is simply Mean Squared Error (MSE) between true added noise $\epsilon$ and predicted noise $\epsilon_\theta$!

Latent Diffusion Models (Stable Diffusion - Rombach et al., 2022)

Denoising a high-resolution $512 \times 512 \times 3$ raw pixel image across 1000 steps requires massive GPU VRAM and compute.

Latent Diffusion Models (LDM / Stable Diffusion) solve this using Perceptual Compression:

  High-Res Image (512x512x3) ──► [ VAE ENCODER ] ──► Latent Representation z_0 (64x64x4)
                                                             │
                                                             ▼
                                      [ RUN DENOISING DIFFUSION IN LATENT SPACE! ]
                                                             │
                                                             ▼
  Generated High-Res Image ◄── [ VAE DECODER ] ◄── Denormalized Latent z_0

1. A Variational Autoencoder (VAE) compresses raw $512 \times 512$ images by $8\times$ down into a compact $64 \times 64 \times 4$ latent space $z_0$.
2. Diffusion runs entirely inside the $64 \times 64$ latent space, cutting GPU memory overhead by 64 times!
3. A VAE Decoder maps the denoised latent vector back into a crisp $512 \times 512$ pixel image.

Conditioning & Classifier-Free Guidance (CFG)

How do we force a Diffusion Model to generate a specific text prompt ("A astronaut riding a horse on Mars")?

1. Encode prompt using a Text Encoder (CLIP / T5) into text embedding vectors $c$.
2. Inject text embeddings into the U-Net denoiser using Cross-Attention layers.
3. Classifier-Free Guidance (CFG): Blends un-conditioned noise predictions with text-conditioned predictions to control prompt adherence:

$$\tilde{\epsilon}\theta(x_t, c) = (1 + w) \epsilon\theta(x_t, c) - w \epsilon_\theta(x_t, \emptyset)$$

Setting guidance scale $w = 7.5$ forces the generator to follow the text prompt strictly.

Say this out loud

Diffusion Models generate data by reversing a gradual Gaussian noise process. The forward process incrementally adds noise over T steps, while a U-Net neural network learns to predict and subtract noise step by step using MSE loss. Stable Diffusion runs the noise process inside a compressed VAE latent space, using Classifier-Free Guidance to enforce text prompt adherence.

Followups to expect

1. What are DDIM Samplers (Denoising Diffusion Implicit Models)? Non Markovian sampling algorithms that allow skipping diffusion steps, reducing generation time from 1000 steps down to 20 or 50 steps without retraining the model.
2. What are Diffusion Transformers (DiT - Peebles & Xie, 2023)? Replacing the standard U-Net backbone with a Transformer Encoder operating on latent image patches, powering modern generation engines like Sora and Flux 1.