5.3 DDPM: the forward and reverse processes
A diffusion model destroys data with a fixed recipe of small Gaussian steps, learns to predict the noise that was added, and generates by running the recipe backwards from pure static, one network call per step.
01The forward process: add noise until it is static
Take a picture and add a faint layer of random static. It still looks like the picture. Add another faint layer, and another. After a few hundred layers the picture is gone and only static is left. That is the whole forward process of a diffusion model, and nothing in it is learned. The model's job is the opposite one: given a slightly noisier version, produce a slightly cleaner one. Chain a thousand of those small clean-ups, starting from pure static, and you have generated a new picture.
In symbols, one step takes the previous noisy version , shrinks it slightly and adds fresh Gaussian noise whose variance is , a small number fixed in advance for each step :
The viewport runs this chain on 2D data so you can watch every point. The data is a swiss roll: 48 small Gaussians (width 0.09) along a spiral, with 400 samples drawn on the front slice at . Time runs to the right; the translucent slices sit at , and each fills with a faint copy of the cloud once the process passes it. The bright slice is the current , the thin lines follow 20 of the points from slice to slice, and colour marks where on the spiral each point started (pale for the inner end, emerald for the outer). The emerald curve above is , the subject of step 2.
Stop the forward chain after 100 of its 1000 steps.
Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 2, eq. (2)
Maths, CS & AI Compendium, Henry Ndubuaku
02Jump to any t: q(x_t | x_0) in closed form
Running 500 steps to get one sample at would make training slow. It is unnecessary: a sum of independent Gaussians is Gaussian, so the whole chain from to collapses into a single step.
- 1
One step, rewritten with α.
With the linear schedule, the signal coefficient and the noise coefficient are: at , 1 and 0; at , , so 0.724 and 0.690; at , , so 0.280 and 0.960; at , , so 0.006 and 1.000. The two are equal near . By the data contributes less than a third of each coordinate, and at it is invisible.
Jump straight to and hold.
Now , where the linear schedule has .
Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 2, eq. (4)
03Training: add known noise, predict it
The generator runs the chain backwards with a learned Gaussian step . That is a good model when is small: the reverse of a tiny Gaussian step is itself almost Gaussian. Training maximises a lower bound on , as in the VAE of lesson 5.1, and the bound reduces to a remarkably plain regression.
- 1
The variational bound splits into one KL term per step. Conditioning on x₀ makes the forward posterior tractable.
This is lesson 5.2's denoising score matching in different units. With , the DSM target for the noisy point is , so a score model and a noise model are related by , and : is DSM summed over noise levels with weight , the weighting Song and Ermon also chose. Predicting , whose size is 1 at every , keeps the regression targets on one scale.
x₀, ε, x_t and ε_θ(x_t, t) all have the image's shape. t has shape (B,): one random timestep per example, fed to the network as an embedding, so a single network learns every noise level.
Hold the forward process at , then hover in the loss above.
Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 3.2 to 3.4, Algorithm 1
Score-Based Generative Modeling through SDEs, Song et al., 2021, Sec. 2 and 3
The rest of this lesson is in Pro
- 04Ancestral sampling, step by step
- 05Noise schedules: linear against cosine
- 06Fast samplers: DDIM, fewer steps, and the cost of sampling
- Python challenge: Noise schedules, the closed-form forward sample and one ancestral step
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What are the forward and reverse processes in a diffusion model?
- The forward process is a fixed recipe that adds a small amount of Gaussian noise to the data at each of T steps until only noise is left; it has no parameters. The reverse process is a learned model that removes a little noise per step. Generation starts from pure noise and applies the reverse step T times, one network evaluation per step.
- Why do diffusion models predict the noise instead of the clean image?
- The variational bound on the likelihood reduces to matching the mean of each reverse step, and writing that mean through the added noise turns training into a regression on the noise. The noise has the same scale at every timestep, which keeps the regression well conditioned, and predicting it is equivalent to predicting the score of the noisy data, up to a factor of minus the noise standard deviation.
- How many sampling steps does a diffusion model need?
- Training typically uses 1000 timesteps, but a trained model can be sampled on a subsequence. Deterministic samplers such as DDIM give good results with roughly 50 to 100 steps, and quality drops noticeably below about 10 to 20 steps unless the model is distilled or a more accurate ODE solver is used. Each step is one network call, so the step count sets latency almost linearly.
Next in Generative AI and Diffusion
5.4Classifier-free guidance and latent diffusionSteering samples with conditioning, and running diffusion in a compressed latent space for speed.Sources
- Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020
- Score-Based Generative Modeling through SDEs, Song et al., 2021
- Maths, CS & AI Compendium, Henry Ndubuaku