Skip to content
LearnMLByDoing
Free

5.3 DDPM: the forward and reverse processes

42 minProadvancedDDPMSongCompendium

A diffusion model destroys data with a fixed recipe of small Gaussian steps, learns to predict the noise that was added, and generates by running the recipe backwards from pure static, one network call per step.

01The forward process: add noise until it is static

Take a picture and add a faint layer of random static. It still looks like the picture. Add another faint layer, and another. After a few hundred layers the picture is gone and only static is left. That is the whole forward process of a diffusion model, and nothing in it is learned. The model's job is the opposite one: given a slightly noisier version, produce a slightly cleaner one. Chain a thousand of those small clean-ups, starting from pure static, and you have generated a new picture.

Loading figure
A one-row image of 64 pixels. Press Step to add one dose of noise, or Run to watch 60 of them. The measured correlation with the original tracks 0.95 t\sqrt{0.95^{\,t}}: 0.77 after 10 steps, 0.28 after 50, where the row looks like static. The spread stays near 1 because each step also shrinks the row by 1−β\sqrt{1-\beta}; switch the shrink off and the spread grows like 1+0.05 t\sqrt{1 + 0.05\,t} instead.

In symbols, one step takes the previous noisy version xt−1x_{t-1}, shrinks it slightly and adds fresh Gaussian noise whose variance is βt\tok{beta}{\beta_t}, a small number fixed in advance for each step t=1,…,Tt = 1, \dots, T:

xt=1−βt  xt−1+βt  εt,εt∼N(0,I),q(xt∣xt−1)=N(1−βt xt−1, βtI)\tok{xt}{x_t} = \sqrt{1-\tok{beta}{\beta_t}}\;x_{t-1} + \sqrt{\tok{beta}{\beta_t}}\;\tok{eps}{\varepsilon_t},\qquad \tok{eps}{\varepsilon_t}\sim\mathcal N(0, I),\qquad q(x_t\mid x_{t-1}) = \mathcal N\big(\sqrt{1-\beta_t}\,x_{t-1},\ \beta_t I\big)
One forward step. Ho et al. use T = 1000 and β rising linearly from 0.0001 to 0.02.

The viewport runs this chain on 2D data so you can watch every point. The data x0\tok{x0}{x_0} is a swiss roll: 48 small Gaussians (width 0.09) along a spiral, with 400 samples drawn on the front slice at t=0t = 0. Time runs to the right; the translucent slices sit at t=0,200,…,1000t = 0, 200, \dots, 1000, and each fills with a faint copy of the cloud once the process passes it. The bright slice is the current xt\tok{xt}{x_t}, the thin lines follow 20 of the points from slice to slice, and colour marks where on the spiral each point started (pale for the inner end, emerald for the outer). The emerald curve above is αˉt\tok{alphabar}{\bar\alpha_t}, the subject of step 2.

Stop the forward chain after 100 of its 1000 steps.

Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 2, eq. (2)

Maths, CS & AI Compendium, Henry Ndubuaku

02Jump to any t: q(x_t | x_0) in closed form

Running 500 steps to get one sample at t=500t = 500 would make training slow. It is unnecessary: a sum of independent Gaussians is Gaussian, so the whole chain from x0x_0 to xtx_t collapses into a single step.

Composing two Gaussian steps1/4
  1. 1
    xt=αt xt−1+1−αt εt,αt=1−βtx_t = \sqrt{\alpha_t}\,x_{t-1} + \sqrt{1-\alpha_t}\,\varepsilon_t,\qquad \alpha_t = 1-\beta_t

    One step, rewritten with α.

q(xt∣x0)=N(αˉt  x0, (1−αˉt) I)⟺xt=αˉt x0+1−αˉt  ε\tok{q}{q(x_t\mid x_0)} = \mathcal N\big(\sqrt{\tok{alphabar}{\bar\alpha_t}}\;\tok{x0}{x_0},\ (1-\tok{alphabar}{\bar\alpha_t})\,I\big)\quad\Longleftrightarrow\quad \tok{xt}{x_t} = \sqrt{\tok{alphabar}{\bar\alpha_t}}\,\tok{x0}{x_0} + \sqrt{1-\tok{alphabar}{\bar\alpha_t}}\;\tok{eps}{\varepsilon}
The forward marginal, and the same thing written as a sample (the reparameterisation: mean plus standard deviation times a standard normal).

With the linear schedule, the signal coefficient αˉt\sqrt{\bar\alpha_t} and the noise coefficient 1−αˉt\sqrt{1-\bar\alpha_t} are: at t=0t = 0, 1 and 0; at t=250t = 250, αˉ=0.524\bar\alpha = 0.524, so 0.724 and 0.690; at t=500t = 500, αˉ=0.079\bar\alpha = 0.079, so 0.280 and 0.960; at t=1000t = 1000, αˉ=0.00004\bar\alpha = 0.00004, so 0.006 and 1.000. The two are equal near t=260t = 260. By t=500t = 500 the data contributes less than a third of each coordinate, and at t=1000t = 1000 it is invisible.

Jump straight to t=250t = 250 and hold.

Now t=500t = 500, where the linear schedule has αˉ=0.079\bar\alpha = 0.079.

Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 2, eq. (4)

03Training: add known noise, predict it

The generator runs the chain backwards with a learned Gaussian step pθ(xt−1∣xt)=N(μθ(xt,t), σt2I)p_\theta(x_{t-1}\mid x_t) = \mathcal N(\mu_\theta(x_t, t),\ \sigma_t^2 I). That is a good model when βt\beta_t is small: the reverse of a tiny Gaussian step is itself almost Gaussian. Training maximises a lower bound on log⁡pθ(x0)\log p_\theta(x_0), as in the VAE of lesson 5.1, and the bound reduces to a remarkably plain regression.

From the ELBO to the simple loss (sketch)1/5
  1. 1
    −log⁡pθ(x0)≤Eq[∑t>1DKL(q(xt−1∣xt,x0) ∥ pθ(xt−1∣xt))]+terms at t=1 and T-\log p_\theta(x_0) \le \mathbb E_q\Big[\textstyle\sum_{t>1} D_{\mathrm{KL}}\big(q(x_{t-1}\mid x_t, x_0)\,\big\|\,p_\theta(x_{t-1}\mid x_t)\big)\Big] + \text{terms at } t = 1 \text{ and } T

    The variational bound splits into one KL term per step. Conditioning on x₀ makes the forward posterior tractable.

Lsimple(θ)=E ∥ε−εθ(xt,t)∥2,xt=αˉt x0+1−αˉt ε,t∼U{1..T}\mathcal L_{\text{simple}}(\theta) = \mathbb E\,\big\|\tok{eps}{\varepsilon} - \tok{eps_model}{\varepsilon_\theta(x_t, t)}\big\|^2,\qquad \tok{xt}{x_t} = \sqrt{\tok{alphabar}{\bar\alpha_t}}\,\tok{x0}{x_0} + \sqrt{1-\tok{alphabar}{\bar\alpha_t}}\,\tok{eps}{\varepsilon},\quad t\sim\mathcal U\{1..T\}
One training step: pick a data point, a random t and a random ε; form x_t in closed form; regress ε_θ onto ε.
Loading figure
You play εθ\varepsilon_\theta. The clean signal is a bump of unknown height and position; you see only xtx_t (cyan) and set a guess x^0\hat x_0. Your noise guess is ε^=(xt−αˉt x^0)/1−αˉt\hat\varepsilon = (x_t - \sqrt{\bar\alpha_t}\,\hat x_0)/\sqrt{1-\bar\alpha_t}, and the loss is exactly SNRt⋅(x0−x^0)2‾\mathrm{SNR}_t \cdot \overline{(x_0 - \hat x_0)^2} with SNRt=αˉt/(1−αˉt)\mathrm{SNR}_t = \bar\alpha_t/(1-\bar\alpha_t). At t=50t = 50 (SNR 33.5) a small error in the bump costs a lot; at t=800t = 800 (SNR 0.0015) every guess scores nearly the same, because ε^≈xt\hat\varepsilon \approx x_t is already the right answer.

This is lesson 5.2's denoising score matching in different units. With σt=1−αˉt\sigma_t = \sqrt{1-\bar\alpha_t}, the DSM target for the noisy point is −ε/σt-\varepsilon/\sigma_t, so a score model and a noise model are related by εθ=−σt sθ\tok{eps_model}{\varepsilon_\theta} = -\sigma_t\, s_\theta, and ∥ε−εθ∥2=σt2 ∥sθ+ε/σt∥2\|\varepsilon - \varepsilon_\theta\|^2 = \sigma_t^2\,\|s_\theta + \varepsilon/\sigma_t\|^2: Lsimple\mathcal L_{\text{simple}} is DSM summed over noise levels with weight σt2\sigma_t^2, the weighting Song and Ermon also chose. Predicting ε\varepsilon, whose size is 1 at every tt, keeps the regression targets on one scale.

one training batch[,]

x₀, ε, x_t and ε_θ(x_t, t) all have the image's shape. t has shape (B,): one random timestep per example, fed to the network as an embedding, so a single network learns every noise level.

Hold the forward process at t=300t = 300, then hover εθ(xt,t)\tok{eps_model}{\varepsilon_\theta(x_t, t)} in the loss above.

Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 3.2 to 3.4, Algorithm 1

Score-Based Generative Modeling through SDEs, Song et al., 2021, Sec. 2 and 3

The rest of this lesson is in Pro

  • 04Ancestral sampling, step by step
  • 05Noise schedules: linear against cosine
  • 06Fast samplers: DDIM, fewer steps, and the cost of sampling
  • Python challenge: Noise schedules, the closed-form forward sample and one ancestral step
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What are the forward and reverse processes in a diffusion model?
The forward process is a fixed recipe that adds a small amount of Gaussian noise to the data at each of T steps until only noise is left; it has no parameters. The reverse process is a learned model that removes a little noise per step. Generation starts from pure noise and applies the reverse step T times, one network evaluation per step.
Why do diffusion models predict the noise instead of the clean image?
The variational bound on the likelihood reduces to matching the mean of each reverse step, and writing that mean through the added noise turns training into a regression on the noise. The noise has the same scale at every timestep, which keeps the regression well conditioned, and predicting it is equivalent to predicting the score of the noisy data, up to a factor of minus the noise standard deviation.
How many sampling steps does a diffusion model need?
Training typically uses 1000 timesteps, but a trained model can be sampled on a subsequence. Deterministic samplers such as DDIM give good results with roughly 50 to 100 steps, and quality drops noticeably below about 10 to 20 steps unless the model is distilled or a more accurate ODE solver is used. Each step is one network call, so the step count sets latency almost linearly.

Next in Generative AI and Diffusion

5.4Classifier-free guidance and latent diffusionSteering samples with conditioning, and running diffusion in a compressed latent space for speed.

Sources

  • Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020
  • Score-Based Generative Modeling through SDEs, Song et al., 2021
  • Maths, CS & AI Compendium, Henry Ndubuaku