1.2 Gradients, Jacobians and the chain rule
A derivative is the best linear approximation of a function at a point: the gradient, the Jacobian and the chain rule are three views of that one idea, and backprop is the chain rule evaluated from the loss backwards.
01The gradient: best linear model of a scalar
Start with a landscape that has two inputs, and , and one output, its height . Pick a point and freeze . What is left is an ordinary curve in , and its slope at the point is the partial derivative . Freeze instead and walk along : that slope is . Lesson 0.5 did this on simple bowls; the widget does it on this lesson's surface.
The surface is a loss over a two-dimensional input, , where is the small two-layer tanh network you will take apart in step 4. Its minimum sits in the valley near . The probe, the emerald dot, is at , where . You can drag it across the surface in the viewport.
Zoom in far enough on any smooth surface and it looks like a plane: the grey tangent plane at the probe. A plane through is fixed by its two slopes, and those slopes are the partial derivatives. Stacked into a vector they form the gradient:
Moving a small distance along a unit direction changes by about . That rate is the directional derivative , and it is the slope of the emerald line on the tangent plane, drawn along the rotating emerald arrow . The white curve is the surface itself along ; near the probe the two agree.
- 1
φ is the angle between u and ∇L, and ‖u‖ = 1.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.1 and 5.2
Deep Learning, Goodfellow, Bengio & Courville, Ch. 4.3
02The Jacobian: a local linear map
A layer of a network is not scalar-valued: it maps a vector to a vector. The viewport now shows a map , the swirl, which rotates every point by radians: a lot near the centre, almost nothing at the edges. The animation morphs the grid from the identity to and back.
The dashed square is a small patch of inputs around with side . Its image , in grey, is curved. The emerald parallelogram is what the best linear approximation predicts:
As in lesson 1.1, a linear map is fixed by where it sends and . The difference is that here the matrix changes from point to point. Each column is also a velocity: is the tangent of the curve that the gridline through in the direction becomes. Hover to trace both gridlines.
2 × 2 here. A layer from 512 features to 128 has a 128 × 512 Jacobian; its transpose is what carries gradients backwards.
Shrink the probe square to a quarter of its side, , at the same point.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.3
The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018, Sec. 4.1, generalization of the Jacobian
03The determinant: local area scaling
Lesson 1.1 showed that a matrix scales every area by and that a negative determinant flips orientation. A nonlinear map does the same locally, with its Jacobian in place of : a tiny square at comes out with area multiplied by . The sheet is coloured by : teal where orientation is kept, brighter for more stretch; rose where it flips; dark where the map collapses area to nothing.
The map is now a fold: a residual layer with a single tanh unit, with and . Its Jacobian is the identity plus a rank-one term, with , and the matrix determinant lemma, (a rank-one change of the identity moves its determinant by one dot product), gives . That is negative inside the band , where the layer pushes points back past their neighbours and the sheet folds over itself in a Z: the rose ribbon. At the probe, the origin, and . The measured ratio is at and at , and the patch fill turns rose because its orientation is reversed.
Switch the map to the swirl, with a large probe square, .
Switch to polar coordinates and move the probe to , .
This is the factor in the change-of-variables formula. If is invertible and has density , then : probability mass is conserved, so density falls where area grows. Normalising flows train invertible layers whose Jacobian determinants are cheap to compute for exactly this reason, and in a polar integral is the same factor.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.3 and 6.7
Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.12
The rest of this lesson is in Pro
- 04The chain rule is matrix multiplication
- 05Forward mode, reverse mode and backprop
- 06Matrix-calculus identities for ML
- 07Gradient checking with central differences
- Python challenge: Numerical Jacobians, JVPs, VJPs and gradient checks
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between a gradient and a Jacobian?
- A gradient belongs to a scalar-valued function of several variables: it is the vector of partial derivatives and points in the direction of steepest increase. A Jacobian belongs to a vector-valued function: it is the matrix with one row per output and one column per input, and each row is the gradient of one output. For a scalar function the Jacobian is a single row, the transposed gradient.
- Why does backpropagation use reverse-mode automatic differentiation?
- Reverse mode computes one row of the Jacobian, a vector-Jacobian product, per backward pass, while forward mode computes one column per pass. A training loss has one output and up to billions of inputs, so reverse mode gets the entire gradient in a single backward pass that costs a small constant times the forward pass. The trade-off is memory, because intermediate activations must be kept for the backward pass.
- How do I check that my gradients are correct?
- Compare the analytic gradient with central finite differences, (f(x + h e_i) minus f(x minus h e_i)) divided by 2h, in float64 with h around 1e-5. Compute the relative error as the norm of the difference divided by the sum of the two norms. Values below about 1e-7 indicate a correct gradient; 1e-4 or more points to a bug. Avoid evaluating exactly at kinks such as ReLU at zero, and check a sample of coordinates because each one costs two forward passes.
Next in Mathematical Foundations
1.3Probability, Gaussians and maximum likelihoodDensities as landscapes: covariance ellipsoids, conditioning, and how least squares falls out of a Gaussian likelihood.Sources
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Deep Learning, Goodfellow, Bengio & Courville
- Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018
- Maths, CS & AI Compendium, Henry Ndubuaku
- The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018