Skip to content
LearnMLByDoing
Free

5.4 Classifier-free guidance and latent diffusion

34 minProadvancedDDPMSongCFGCompendiumVAELDMADM

Classifier-free guidance pushes every denoising step away from what the model would draw without the prompt, buying prompt fidelity with diversity, and latent diffusion runs that loop on a compressed image 48 times smaller than the pixels.

01Conditional generation: from p(x) to p(x | c)

Lessons 5.2 and 5.3 built a sampler for p(x)p(x): start from noise and follow a learned field back to the data. A product needs more: a given prompt, a given class. The model becomes conditional, εθ(xt,t,c)\tok{eps_c}{\varepsilon_\theta(x_t, t, c)}, trained exactly like DDPM but with the label or caption c\tok{cond}{c} as an extra input, and the same sampler then draws from p(x∣c)p(x\mid c).

The viewport is that loop on a toy you can see whole. The floor holds two classes of 2D data, 220 real points each: an emerald ring (c=0c = 0, 12 Gaussian blobs of spread 0.25 on a circle of radius 1.1 around (−0.5, 0.3)(-0.5,\,0.3)) and a cyan bar (c=1c = 1, 10 blobs from (−2, −1.2)(-2,\,-1.2) to (2, 0.9)(2,\,0.9)). The bar cuts through the ring. There is no network: every ε\varepsilon is the exact one of this mixture under DDPM's noising (linear schedule, T=1000T = 1000), so what you see is the method itself, not a training artefact.

The 300 white particles xt\tok{xt}{x_t} start as pure noise xT∼N(0,I)x_T\sim\mathcal N(0, I) on the top plane (t=999t = 999) and come down one deterministic DDIM step at a time (lesson 5.3); their grey trails show the paths. On landing each turns emerald if the exact classifier calls it a ring, cyan if it looks more like a bar, and rose if it lies off the data (density below that of 95 % of real ring points). The amber arrows are the field the sampler follows, −ε/σ-\varepsilon/\sigma, at the current tt (they stop at t=120t = 120, below which the field turns into the spiky score of the raw data); the grey and emerald layers under them are the unconditional and conditional predictions.

Right now w=0w = 0: plain conditional sampling of the ring. The HUD reads bleed 18.3 %: almost a fifth of the samples land where the bar is the more likely class, mostly around the two crossings, and 11.9 of 12 blobs are used. These are faithful draws from p(x∣ring)p(x\mid\text{ring}); the classes overlap, so some ring samples look like bars. Real conditional models bleed more than this, because a network learns to use its conditioning input only partly.

Set w=−1w = -1. Under this lesson's convention that makes the guided prediction equal to the unconditional one.

Condition on c=1c = 1 at w=0w = 0.

Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020, Sec. 3.2, Algorithm 2

Score-Based Generative Modeling through SDEs, Song et al., 2021, Sec. 5

02Classifier guidance: Bayes' rule on scores

Before classifier-free guidance there was classifier guidance (Dhariwal and Nichol, 2021). Take an unconditional diffusion model and a separate classifier, and steer the sampler with the classifier's gradient. Bayes' rule says why that works, and it says it in score language, where the awkward normaliser p(c)p(c) disappears:

Bayes on scores1/5
  1. 1
    pt(x∣c)=pt(c∣x) pt(x)p(c)p_t(x\mid c) = \frac{p_t(c\mid x)\,p_t(x)}{p(c)}

    Bayes' rule for the noisy data at step t. The prior p(c) is a constant: 1/2 for each class here.

Read the last line as two forces. The diffusion model pulls toward data of any class; the classifier pushes toward regions it is confident belong to cc. Hover ∇xlog⁡pt(c∣x)\tok{clf}{\nabla_x\log p_t(c\mid x)} and the viewport draws this push as a white layer (drawn 6× longer): almost zero on the parts of the ring far from the bar, where a sample is clearly a ring, and large on and around the bar, where a ring sample would look like the wrong class. Its size is 1−pt(c∣x)1 - p_t(c\mid x) times the gradient of the log-odds, so it fades exactly where the classifier is already sure.

Double the classifier term. With exact models, γ=2\gamma = 2 on the unconditional model is the same field as w=1w = 1 in the next step's formula, so this sets w=1w = 1.

Score-Based Generative Modeling through SDEs, Song et al., 2021, Sec. 5 (controllable generation)

Classifier-Free Diffusion Guidance, Ho & Salimans, 2022, Sec. 3.1

03Classifier-free guidance: one network, two predictions

Ho and Salimans (2022) noticed that the classifier is already hiding inside a conditional model. Train one network on both tasks by replacing the condition with a null token ∅\varnothing at random, typically for 10 to 20 % of training examples (condition dropout). The loss is DDPM's, ∥ε−εθ(xt,t,c)∥2\|\varepsilon - \varepsilon_\theta(x_t, t, c)\|^2, with cc sometimes set to ∅\varnothing. The same weights then give εθ(xt,t,c)\tok{eps_c}{\varepsilon_\theta(x_t,t,c)} and εθ(xt,t,∅)\tok{eps_u}{\varepsilon_\theta(x_t,t,\varnothing)}, and Bayes' rule turns their difference into a classifier:

From the implicit classifier to the guided prediction1/5
  1. 1
    ∇xlog⁡pt(c∣x)=∇xlog⁡pt(x∣c)−∇xlog⁡pt(x)\nabla_x \log p_t(c\mid x) = \nabla_x \log p_t(x\mid c) - \nabla_x \log p_t(x)

    Bayes on scores, rearranged: the classifier's gradient is the difference of two scores.

Loading figure
Drag the tips of εc\varepsilon_c and εu\varepsilon_u and the slider. Every guided prediction lies on the dashed line through εu\varepsilon_u (w=−1w = -1) and εc\varepsilon_c (w=0w = 0); w>0w > 0 extrapolates past εc\varepsilon_c, away from εu\varepsilon_u, by ww copies of their difference. The slider is linked to the viewport's ww. The toggle swaps εu\varepsilon_u for a negative prompt's prediction, step 6.

The viewport is at w=3w = 3. Late in each run, on the upper-left of the ring, far from the bar, the grey and emerald arrows coincide, so the amber one does too: guidance changes nothing where the classes do not compete. Near the crossings they differ and the amber arrow is pushed past the emerald one, away from the bar. The result: bleed 1.7 %, down from 18.3 % at w=0w = 0, but only 5.9 of 12 blobs used. The samples crowd onto the upper-left arc of the ring, the side away from the bar.

one guided denoising step (batched)[,,,]

Implementations stack the conditional and null-prompt copies of z_t into one batch, run the network once, split the output into ε_c and ε_u, and combine. Guidance doubles the compute per step; it does not add a model.

Classifier-Free Diffusion Guidance, Ho & Salimans, 2022, Sec. 3.2, Algorithms 1 and 2

The rest of this lesson is in Pro

  • 04The fidelity–diversity trade-off, and over-saturation
  • 05Latent diffusion: compress first, then diffuse
  • 06In practice: guidance scale, steps, negative prompts and seeds
  • Python challenge: Guided noise, the implicit classifier and the latent saving
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is classifier-free guidance in diffusion models?
Classifier-free guidance trains one diffusion network both with and without its conditioning, by dropping the prompt for a random fraction of training examples. At sampling time it predicts the noise twice per step, with the prompt and with an empty prompt, and extrapolates from the second toward and past the first. The difference of the two predictions acts as the gradient of an implicit classifier, so no separate classifier is needed.
What does the CFG scale or guidance scale do in Stable Diffusion?
It sets how far each denoising step extrapolates away from the unconditional prediction toward the prompt-conditioned one. Higher values make images follow the prompt more closely but reduce variety, and very high values produce over-saturated, distorted images. A scale of 1 means no guidance; values around 5 to 9 are common, with 7.5 a frequent default.
Why does Stable Diffusion work in latent space instead of pixel space?
Because an autoencoder can compress a 512 by 512 by 3 image to a 64 by 64 by 4 latent that keeps the perceptually important content. Running the denoising network there processes 48 times fewer values per step, and attention over spatial positions becomes thousands of times cheaper. The decoder turns the final latent back into pixels once, after the last step.

Sources

  • Denoising Diffusion Probabilistic Models, Ho, Jain & Abbeel, 2020
  • Score-Based Generative Modeling through SDEs, Song et al., 2021
  • Classifier-Free Diffusion Guidance, Ho & Salimans, 2022
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Auto-Encoding Variational Bayes, Kingma & Welling, 2014
  • High-Resolution Image Synthesis with Latent Diffusion Models, Rombach et al., 2022
  • Diffusion Models Beat GANs on Image Synthesis, Dhariwal & Nichol, 2021