Skip to content
LearnMLByDoing
Free

3.3 Initialisation, normalisation and training stability

32 minProintermediateDLKarpathyD2LHOMLRaschkaVaswani

Every layer multiplies the signal's variance by roughly n·Var(w), so a deep stack fades to zero, saturates or explodes unless initialisation and normalisation hold that factor near 1.

01Why deep stacks misbehave

The viewport holds a 16-layer MLP with 256 units per layer and no biases, evaluated on one fixed batch XX of 64 standardised rows. Each translucent curtain is the histogram of one layer's 64×256=16,38464 \times 256 = 16{,}384 activations h(l)\tok{act}{h^{(l)}}, with layer 1 in front. The wall at the left carries two series on a log scale: the emerald curve is each layer's standard deviation, and the amber bars show the backward signal. A wave carries the signal through the stack, then back.

Start with one number. Suppose each layer multiplies the size of its signal by the same factor gg. After one layer the signal is gg, after two g⋅g=g2g \cdot g = g^2, after sixteen g16g^{16}. With g=0.5g = 0.5 that is 0.516=0.0000150.5^{16} = 0.000015; with g=2g = 2 it is 65,53665{,}536. Only g=1g = 1 keeps the signal the size it started. Drag gg below and above 1 and watch the decades pile up.

Loading figure
The curve starts at g=0.16g = 0.16, the factor of this step's small init: 0.1616=1.9×10−130.16^{16} = 1.9 \times 10^{-13}, about 13 decades lost. The dashed curves are the factors of two inits you meet in step 3.
z(l)=h(l−1) W(l),h(l)=σ(z(l)),h(0)=Xz^{(l)} = \tok{act}{h^{(l-1)}}\,W^{(l)}, \qquad \tok{act}{h^{(l)}} = \sigma\big(z^{(l)}\big), \qquad h^{(0)} = X
Row-vector convention, as in lesson 3.1. Weights are drawn i.i.d. with mean 0 and variance Var(w).
W(l)[,]65,536 values

one per layer

h(l)[,]16,384 values

one curtain in the viewport

Variance through one layer1/5
  1. 1
    zj=∑i=1ninhi wijz_j = \sum_{i=1}^{\tok{fan_in}{n_{\text{in}}}} h_i\, w_{ij}

    Every pre-activation is a sum of n_in products.

Loading figure
The derivation checked by sampling: 4,000 draws of one unit's zz. Whatever nn and init you pick, the measured variance matches n Var(w) E[h2]n\,\mathrm{Var}(w)\,\mathbb{E}[h^2] to within a few percent. The init buttons also set the 3D scene.

The viewport uses the classic small init w∼N(0,0.012)w \sim \mathcal{N}(0, 0.01^2) with tanh. Then ninVar(w)=256⋅10−4=0.0256n_{\text{in}}\mathrm{Var}(w) = 256 \cdot 10^{-4} = 0.0256, so each layer should shrink the std by 0.0256=0.16\sqrt{0.0256} = 0.16. The measurement agrees: std h(1)=0.156h^{(1)} = 0.156, h(2)=0.0247h^{(2)} = 0.0247 (a ratio of 0.158), and by layer 16 the std is 1.8×10−131.8 \times 10^{-13}: close to 0.160.16 multiplied by itself sixteen times. From layer 2 on, every curtain is a needle at 0 and its colour fades to slate. The wall's log scale stops at 10±610^{\pm 6}, so the emerald curve runs into the floor of the wall, and a chip there gives the true value. Hover Var(w)\tok{var_w}{\mathrm{Var}(w)} to draw the prediction beside the measured curve.

δ(l)=(δ(l+1) W(l+1)⊤)⊙σ′(z(l))    ⇒    Var(δ(l))≈nout Var(w)  E[σ′(z)2]  Var(δ(l+1))\tok{delta}{\boldsymbol{\delta}^{(l)}} = \big(\boldsymbol{\delta}^{(l+1)}\,W^{(l+1)\top}\big) \odot \sigma'\big(z^{(l)}\big) \;\;\Rightarrow\;\; \mathrm{Var}\big(\boldsymbol{\delta}^{(l)}\big) \approx n_{\text{out}}\,\mathrm{Var}(w)\;\mathbb{E}\big[\sigma'(z)^2\big]\;\mathrm{Var}\big(\boldsymbol{\delta}^{(l+1)}\big)
The backward pass from lesson 3.1. The gradient that reaches layer 1 is a product of L − 1 Jacobian factors of this size.

Backpropagation multiplies by the same weights, transposed, so it inherits the same factor. Near 0, tanh⁡′≈1\tanh' \approx 1 and the backward factor is again 0.0256 per layer: the amber bars show ∥δ(1)∥/∥δ(16)∥=1.2×10−12\|\boldsymbol{\delta}^{(1)}\| / \|\boldsymbol{\delta}^{(16)}\| = 1.2 \times 10^{-12}. The first layer receives no usable gradient. This is the vanishing gradient problem, and its mirror image, factors above 1, is the exploding gradient problem.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.2.5 and 8.4

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, makemore part 3: activations and gradients

02Xavier initialisation for tanh

The fix is to choose Var(w)\mathrm{Var}(w) so that the per-layer factor is 1. The forward pass sums over ninn_{\text{in}} inputs and the backward pass sums over noutn_{\text{out}} outputs, so the two directions ask for different variances. Glorot and Bengio split the difference.

Balancing the forward and backward pass1/5
  1. 1
    tanh⁡(z)≈z,tanh⁡′(z)≈1(∣z∣≪1)\tanh(z) \approx z, \qquad \tanh'(z) \approx 1 \quad (|z| \ll 1)

    Linearise around the origin, where a well-initialised tanh network operates.

In the viewport the curtains now keep a wide bell all the way down: std h(1)=0.627h^{(1)} = 0.627, h(2)=0.484h^{(2)} = 0.484, h(16)=0.183h^{(16)} = 0.183, and ∥δ(1)∥/∥δ(16)∥=0.231\|\boldsymbol{\delta}^{(1)}\| / \|\boldsymbol{\delta}^{(16)}\| = 0.231. The slow narrowing is real, not noise: ∣tanh⁡z∣<∣z∣|\tanh z| < |z|, so tanh hands the next layer slightly less variance than it received, and the linear analysis is an upper bound. A fivefold change across 16 layers is benign compared with the 101210^{12} of the previous step.

Loading figure
The scene's two series on separate, unclamped log axes. The buttons set the 3D scene too, so you can compare the four inits here and in the viewport.

Use far too large weights: N(0,1)\mathcal{N}(0, 1), so ninVar(w)=256n_{\text{in}}\mathrm{Var}(w) = 256 and pre-activations have a std near 16.

Sigmoid is worse. Its slope σ′(z)=σ(z)(1−σ(z))\sigma'(z) = \sigma(z)(1 - \sigma(z)) is at most 0.25, reached at z=0z = 0. Even with Xavier weights, which keep n Var(w)=1n\,\mathrm{Var}(w) = 1, every layer multiplies the backward signal by at most 0.25, and fifteen layers give at most 0.2515=9×10−100.25^{15} = 9 \times 10^{-10}. Its outputs also sit around 0.5, not 0, which shifts every next layer's inputs. That is the textbook cause of vanishing gradients and why hidden layers moved to tanh, then ReLU.

Same Xavier weights, sigmoid activation.

Hands-On Machine Learning, Aurélien Géron, Ch. 11: Glorot and He initialization

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 5.4 Numerical stability and initialization

03Kaiming initialisation for ReLU

ReLU breaks the linear assumption in a precise way: it sets every negative pre-activation to 0. Xavier's variance then loses half of the signal at every layer. He et al. redo the bookkeeping with the second moment E[h2]\mathbb{E}[h^2] from the first derivation.

Where the factor 2 comes from1/4
  1. 1
    h=max⁡(0,z),z symmetric around 0h = \max(0, z), \qquad z \text{ symmetric around } 0

    Zero-mean weights make every pre-activation symmetric, so half of them are clipped.

The viewport is set to Kaiming with ReLU. Each curtain is a slate post at 0, the exact zeros (about half of all entries), next to a half-bell, and the shape is the same at every depth: std h(1)=0.829h^{(1)} = 0.829, h(16)=0.853h^{(16)} = 0.853, and ∥δ(1)∥/∥δ(16)∥=1.21\|\boldsymbol{\delta}^{(1)}\| / \|\boldsymbol{\delta}^{(16)}\| = 1.21.

Keep ReLU but go back to Xavier's variance, 1/nin1/n_{\text{in}} instead of 2/nin2/n_{\text{in}}.

Loading figure
The factor 12\tfrac12 measured: with ReLU inputs, E[h2]=12\mathbb{E}[h^2] = \tfrac12, so Xavier's n Var(w)=1n\,\mathrm{Var}(w) = 1 predicts Var(z)=0.5\mathrm{Var}(z) = 0.5. Switch to Kaiming and it is back to 1.

Kaiming sits exactly on the critical point, and the Weight gain slider in the cockpit lets you step off it. It multiplies every weight's std by a constant, so the per-layer factor becomes gain × 1. A gain of 0.9 already costs a factor of 5 in std over 16 layers (0.158 at layer 16), and a gain of 1.1 grows it by a similar factor (3.92): a 10% error in the init scale compounds just like the 0.16 of step 1, only slower.

Kaiming with ReLU, but every weight 10% too large.

A dead ReLU unit has slope 0 on every input, so no gradient ever reaches its weights and it cannot recover (lesson 3.1 shows one dying). The standard fixes change the slope on the negative side: Leaky ReLU uses 0.01z0.01z there, and GELU and SiLU, the activations of most transformers, are smooth with a small non-zero slope for negative inputs. A small positive bias at init, a lower learning rate and normalisation also make dead units rarer.

Same Kaiming weights with Leaky ReLU.

Frameworks expose the choice as a mode. Kaiming's fan-in mode preserves forward variance; its fan-out mode, Var(w)=2/nout\mathrm{Var}(w) = 2/n_{\text{out}}, preserves the backward signal instead. For square layers like these the two coincide.

Hands-On Machine Learning, Aurélien Géron, Ch. 11: Glorot and He initialization

Deep Learning, Goodfellow, Bengio & Courville, Ch. 8.4

The rest of this lesson is in Pro

  • 04BatchNorm: normalise each feature over the batch
  • 05LayerNorm, RMSNorm and where transformers put them
  • 06Residuals, clipping, warm-up and diagnosing NaNs
  • Python challenge: Initialise and normalise a deep stack
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between Xavier and He (Kaiming) initialization?
Both draw zero-mean random weights and differ only in the variance. Xavier (Glorot) uses 2 divided by fan-in plus fan-out, which keeps the signal's variance roughly constant through tanh or linear layers in both the forward and backward pass. He (Kaiming) uses 2 divided by fan-in, twice as large, because ReLU sets half of its inputs to zero and would otherwise halve the variance at every layer. Use He for ReLU-family activations and Xavier for tanh, sigmoid or linear layers.
Should I use BatchNorm or LayerNorm?
BatchNorm normalises each feature over the batch, works well for convolutional networks trained with reasonably large batches, and needs running statistics at inference. LayerNorm normalises each example over its own features, so it does not depend on batch size, behaves the same in training and inference, and is the standard choice for transformers and recurrent networks. If your per-device batch is small, prefer LayerNorm or GroupNorm, or synchronise BatchNorm statistics across devices.
Why does my training loss become NaN?
Usually a value overflowed or an invalid operation occurred: exploding activations or gradients, a learning rate that is too high, the log of zero, or a float16 overflow above 65,504. Check the gradient norm and activation statistics just before the NaN. Typical fixes are a lower learning rate with warm-up, gradient clipping, proper initialisation and normalisation, numerically stable loss functions, and loss scaling for mixed precision.

Next in Deep Learning and Autograd

3.4Convolutions, receptive fields and feature mapsWeight sharing as an inductive bias: kernels sliding over images, stacking into hierarchical receptive fields.

Sources

  • Deep Learning, Goodfellow, Bengio & Courville
  • Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola
  • Hands-On Machine Learning, Aurélien Géron
  • Build a Large Language Model (From Scratch), Sebastian Raschka
  • Attention Is All You Need, Vaswani et al., 2017