Skip to content
LearnMLByDoing
Free

1.2 Gradients, Jacobians and the chain rule

30 minProintermediateMMLDLAD surveyCompendiumParr

A derivative is the best linear approximation of a function at a point: the gradient, the Jacobian and the chain rule are three views of that one idea, and backprop is the chain rule evaluated from the loss backwards.

01The gradient: best linear model of a scalar

Start with a landscape that has two inputs, x1x_1 and x2x_2, and one output, its height LL. Pick a point and freeze x2x_2. What is left is an ordinary curve in x1x_1, and its slope at the point is the partial derivative ∂L/∂x1\partial L/\partial x_1. Freeze x1x_1 instead and walk along x2x_2: that slope is ∂L/∂x2\partial L/\partial x_2. Lesson 0.5 did this on simple bowls; the widget does it on this lesson's surface.

Loading figure
The map is the surface seen from above, as contour lines. The cyan line walks along x1x_1 with x2x_2 frozen, the emerald line along x2x_2. The plot under it (beside it on wide screens) shows the height along each walk. At x0=(0.6, 0.4)\mathbf{x}_0=(0.6,\ 0.4) the cyan slice climbs with slope 0.8730.873 and the emerald one falls with slope −0.358-0.358. Those two numbers are the amber arrow. Drag x0\mathbf{x}_0: the probe in the viewport follows.

The surface is a loss over a two-dimensional input, L(x)=12∥h(x)−t∥2L(\mathbf{x})=\tfrac12\|h(\mathbf{x})-\mathbf{t}\|^2, where hh is the small two-layer tanh network you will take apart in step 4. Its minimum sits in the valley near (−0.59,−0.59)(-0.59,-0.59). The probe, the emerald dot, is at x0=(0.6, 0.4)\mathbf{x}_0=(0.6,\ 0.4), where L=1.405L=1.405. You can drag it across the surface in the viewport.

Zoom in far enough on any smooth surface and it looks like a plane: the grey tangent plane at the probe. A plane through (x0,L(x0))(\mathbf{x}_0, L(\mathbf{x}_0)) is fixed by its two slopes, and those slopes are the partial derivatives. Stacked into a vector they form the gradient:

L(x0+δ)  ≈  L(x0)+∇L(x0)⊤δ,∇L=(∂L/∂x1∂L/∂x2)=(0.873−0.358)L(\mathbf{x}_0+\boldsymbol{\delta})\;\approx\;L(\mathbf{x}_0)+\tok{gradient}{\nabla L(\mathbf{x}_0)}^{\top}\boldsymbol{\delta},\qquad \tok{gradient}{\nabla L}=\begin{pmatrix}\partial L/\partial x_1\\ \partial L/\partial x_2\end{pmatrix}=\begin{pmatrix}0.873\\ -0.358\end{pmatrix}
The first-order Taylor model at the probe. The error is o(‖δ‖): it shrinks faster than δ itself. For a smooth L like this one, halving δ cuts the error to about a quarter.

Moving a small distance tt along a unit direction u\tok{direction}{\mathbf{u}} changes LL by about t ∇L⊤ut\,\nabla L^{\top}\mathbf{u}. That rate is the directional derivative DuL=∇L⋅uD_{\mathbf{u}}L=\nabla L\cdot\mathbf{u}, and it is the slope of the emerald line on the tangent plane, drawn along the rotating emerald arrow u\mathbf{u}. The white curve is the surface itself along u\mathbf{u}; near the probe the two agree.

Why the gradient is the steepest direction1/4
  1. 1
    DuL=∇L⊤u=∥∇L∥ ∥u∥cos⁡φ=∥∇L∥cos⁡φD_{\mathbf{u}}L=\nabla L^{\top}\mathbf{u}=\|\nabla L\|\,\|\mathbf{u}\|\cos\varphi=\|\nabla L\|\cos\varphi

    φ is the angle between u and ∇L, and ‖u‖ = 1.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.1 and 5.2

Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3

02The Jacobian: a local linear map

A layer of a network is not scalar-valued: it maps a vector to a vector. The viewport now shows a map f:R2→R2f:\mathbb{R}^2\to\mathbb{R}^2, the swirl, which rotates every point by 1.6 e−∥x∥2/21.6\,e^{-\|\mathbf{x}\|^2/2} radians: a lot near the centre, almost nothing at the edges. The animation morphs the grid from the identity to ff and back.

The dashed square is a small patch of inputs around x0=(0.6, 0.4)\mathbf{x}_0=(0.6,\ 0.4) with side s=0.6s=0.6. Its image f(□)f(\square), in grey, is curved. The emerald parallelogram is what the best linear approximation predicts:

f(x0+δ)≈f(x0)+Jf(x0) δ,[Jf]ij=∂fi∂xjf(\mathbf{x}_0+\boldsymbol{\delta})\approx f(\mathbf{x}_0)+\tok{jacobian}{J_f(\mathbf{x}_0)}\,\boldsymbol{\delta},\qquad \big[\tok{jacobian}{J_f}\big]_{ij}=\frac{\partial f_i}{\partial x_j}
One row per output, one column per input. For a scalar f the Jacobian is the 1 × n row ∇fᵀ.
Jf(0.6, 0.4)=(0.848−0.5991.0760.419)=( ∂f∂x1 ∣ ∂f∂x2 )\tok{jacobian}{J_f(0.6,\,0.4)}=\begin{pmatrix}0.848&-0.599\\1.076&0.419\end{pmatrix}=\Big(\ \tok{jac_col}{\tfrac{\partial f}{\partial x_1}}\ \Big|\ \tok{jac_col}{\tfrac{\partial f}{\partial x_2}}\ \Big)
Column j is J eⱼ, the image of the j-th basis vector: the amber and cyan arrows.

As in lesson 1.1, a linear map is fixed by where it sends e1\mathbf{e}_1 and e2\mathbf{e}_2. The difference is that here the matrix changes from point to point. Each column is also a velocity: ∂f/∂x1\partial f/\partial x_1 is the tangent of the curve that the gridline through x0\mathbf{x}_0 in the x1x_1 direction becomes. Hover ∂f/∂xj\tok{jac_col}{\partial f/\partial x_j} to trace both gridlines.

J_f for f: ℝⁿ → ℝᵐ[,]

2 × 2 here. A layer from 512 features to 128 has a 128 × 512 Jacobian; its transpose is what carries gradients backwards.

Shrink the probe square to a quarter of its side, s=0.15s=0.15, at the same point.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.3

The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018, Sec. 4.1, generalization of the Jacobian

03The determinant: local area scaling

Lesson 1.1 showed that a matrix scales every area by ∣det⁡A∣|\det A| and that a negative determinant flips orientation. A nonlinear map does the same locally, with its Jacobian in place of AA: a tiny square at x0\mathbf{x}_0 comes out with area multiplied by ∣det⁡Jf(x0)∣|\det J_f(\mathbf{x}_0)|. The sheet is coloured by det⁡J\tok{det}{\det J}: teal where orientation is kept, brighter for more stretch; rose where it flips; dark where the map collapses area to nothing.

area⁡(f(□))  ≈  ∣det⁡Jf(x0)∣  area⁡(□)\operatorname{area}\big(f(\square)\big)\;\approx\;\big|\tok{det}{\det J_f(\mathbf{x}_0)}\big|\;\operatorname{area}(\square)
Exact in the limit s → 0. The HUD shows the measured, signed area ratio of the grey patch next to det J.

The map is now a fold: a residual layer with a single tanh unit, f(x)=x+vtanh⁡(w⊤x)f(\mathbf{x})=\mathbf{x}+\mathbf{v}\tanh(\mathbf{w}^{\top}\mathbf{x}) with w=(1.9, 1.1)\mathbf{w}=(1.9,\ 1.1) and v=(−0.74, −0.43)\mathbf{v}=(-0.74,\ -0.43). Its Jacobian is the identity plus a rank-one term, J=I+(1−t2) vw⊤J=I+(1-t^2)\,\mathbf{v}\mathbf{w}^{\top} with t=tanh⁡(w⊤x)t=\tanh(\mathbf{w}^{\top}\mathbf{x}), and the matrix determinant lemma, det⁡(I+ab⊤)=1+b⊤a\det(I+\mathbf{a}\mathbf{b}^{\top})=1+\mathbf{b}^{\top}\mathbf{a} (a rank-one change of the identity moves its determinant by one dot product), gives det⁡J=1+(1−t2) w⊤v=1−1.879 (1−t2)\det J=1+(1-t^2)\,\mathbf{w}^{\top}\mathbf{v}=1-1.879\,(1-t^2). That is negative inside the band ∣w⊤x∣<0.84|\mathbf{w}^{\top}\mathbf{x}|<0.84, where the layer pushes points back past their neighbours and the sheet folds over itself in a Z: the rose ribbon. At the probe, the origin, J=(−0.406−0.814−0.8170.527)J=\begin{pmatrix}-0.406&-0.814\\-0.817&0.527\end{pmatrix} and det⁡J=−0.879\det J=-0.879. The measured ratio is −0.814-0.814 at s=0.3s=0.3 and −0.862-0.862 at s=0.15s=0.15, and the patch fill turns rose because its orientation is reversed.

Switch the map to the swirl, with a large probe square, s=0.6s=0.6.

Switch to polar coordinates and move the probe to r=0.6r=0.6, θ=0.4\theta=0.4.

This is the factor in the change-of-variables formula. If y=f(x)\mathbf{y}=f(\mathbf{x}) is invertible and x\mathbf{x} has density pXp_X, then pY(y)=pX(x) / ∣det⁡Jf(x)∣p_Y(\mathbf{y})=p_X(\mathbf{x})\,/\,|\det J_f(\mathbf{x})|: probability mass is conserved, so density falls where area grows. Normalising flows train invertible layers whose Jacobian determinants are cheap to compute for exactly this reason, and r dr dθr\,dr\,d\theta in a polar integral is the same factor.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.3 and 6.7

Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.12

The rest of this lesson is in Pro

  • 04The chain rule is matrix multiplication
  • 05Forward mode, reverse mode and backprop
  • 06Matrix-calculus identities for ML
  • 07Gradient checking with central differences
  • Python challenge: Numerical Jacobians, JVPs, VJPs and gradient checks
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between a gradient and a Jacobian?
A gradient belongs to a scalar-valued function of several variables: it is the vector of partial derivatives and points in the direction of steepest increase. A Jacobian belongs to a vector-valued function: it is the matrix with one row per output and one column per input, and each row is the gradient of one output. For a scalar function the Jacobian is a single row, the transposed gradient.
Why does backpropagation use reverse-mode automatic differentiation?
Reverse mode computes one row of the Jacobian, a vector-Jacobian product, per backward pass, while forward mode computes one column per pass. A training loss has one output and up to billions of inputs, so reverse mode gets the entire gradient in a single backward pass that costs a small constant times the forward pass. The trade-off is memory, because intermediate activations must be kept for the backward pass.
How do I check that my gradients are correct?
Compare the analytic gradient with central finite differences, (f(x + h e_i) minus f(x minus h e_i)) divided by 2h, in float64 with h around 1e-5. Compute the relative error as the norm of the difference divided by the sum of the two norms. Values below about 1e-7 indicate a correct gradient; 1e-4 or more points to a bug. Avoid evaluating exactly at kinks such as ReLU at zero, and check a sample of coordinates because each one costs two forward passes.

Next in Mathematical Foundations

1.3Probability, Gaussians and maximum likelihoodDensities as landscapes: covariance ellipsoids, conditioning, and how least squares falls out of a Gaussian likelihood.

Sources

  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Deep Learning, Goodfellow, Bengio & Courville
  • Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018