0.5 Partial derivatives and gradients: walking downhill
A function of two inputs is a landscape, and the gradient is the arrow that tells you which way is uphill, so walking against it, one small step at a time, is how a model learns.
01A function of two inputs is a landscape
In Lesson 0.1 a function took one number and gave back one number, and its graph was a curve. Now the function takes two numbers, and , and still gives back one. Picture the floor of the viewport as a map: every spot on it has an and a , and above each spot the ground is raised to the function's output. The result is a landscape. The one on screen is a round bowl, and the white dot is the point , .
Worked example. The bowl's rule is: square each input, then add. At the white dot that is , so the ground there is 5 units high, and the HUD reads . At the centre, : the bottom of the bowl, marked with a small emerald ring.
The faint rings on the floor are contour lines, like on a hiking map: each one joins points of equal height. The white dashed loop is the contour through your point, so every point on it has height 5. The small white walker follows it round and round, and the HUD's walker readout stays at the whole way. Notice that the rings crowd together further out: crowded contours mean steep ground.
Switch to the two hills landscape and put the probe at , , on the side of a hill.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.2
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: thinking about multivariable functions
02Partial derivatives: slice the landscape and look at one curve
Suppose you are only allowed to walk east and west, along , never north or south. The ground under your feet then traces a single curve. In the viewport the cyan sheet slices the bowl along the line , and the cyan curve is that slice. The slope of that curve at your point is called the partial derivative of with respect to . The amber sheet does the same thing in the other direction: it slices along , for walking along only.
Worked example. Freeze . The bowl becomes , an ordinary one-input curve. From Lesson 0.4, the slope of is , and the is a constant that does not change the slope, so at the slope is . Now freeze instead: , with slope , which is at . The HUD agrees: and . From this spot, walking along is twice as steep as walking along .
Back in the viewport, the small cyan walker slides back and forth along the cyan slice carrying a short tangent line. Watch the tangent tilt as it moves: as the walker passes , the bottom of the slice, the tangent goes flat and the walker readout's changes sign.
Move the probe to the other side of the bowl: , .
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: partial derivatives
03The gradient: both slopes in one arrow
You now have two slopes at one point. Put them together as one arrow: go units in the direction (the cyan leg), then units in the direction (the amber leg). The emerald arrow from the start of the first leg to the end of the second is the gradient. It is the vector addition, tip to tail, from Lesson 0.2.
Worked example. At the two slopes are and , so the gradient is the arrow . Its length, by Pythagoras, is : the HUD's . In the viewport the three arrows lie flat against the bowl at your point, along the two tangent lines, and they are drawn at 0.2 times their true length so that they fit.
One slope per input. A model with a million weights has a gradient with a million entries, one partial slope per weight, and the same rules apply.
Two facts about this arrow are visible in the viewport. First, it points uphill: away from the bottom of the bowl. Second, it crosses the white dashed contour at a right angle. The walker on the contour carries its own small emerald gradient arrow, and wherever the walker is, that arrow sticks straight out of the line. This makes sense: along a contour the height does not change, so all the climbing must happen across it.
Switch to the stretched bowl and move the probe to , .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 5.2
04Why the gradient is the steepest way up
Stand on a hillside and turn slowly on the spot. Facing one way the ground rises in front of you, facing the other way it falls, and in two directions in between it is level: those two run along the contour. The gradient points in the one direction where the ground rises fastest.
Worked example. The slope you feel walking in a direction (an arrow of length 1) is the dot product from Lesson 0.2: multiply matching entries and add. At on the bowl, with :
- 1
Facing along +x you feel exactly ∂f/∂x.
The viewport now shows the gradient field: a small emerald arrow at every spot of a grid, lying on the landscape, each one the gradient there. On the round bowl they all point straight out from the centre, they grow longer further out where the bowl is steeper, and each one crosses the contour rings at a right angle.
Show the field on the two hills and put the probe in the middle, at , .
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Multivariable calculus: gradient and directional derivatives
05Walking downhill: gradient descent
If the gradient points uphill, the opposite arrow, , points downhill, and it is the steepest way down. That gives a recipe for finding the bottom of a landscape you cannot see all at once: look at the slope under your feet, take a small step the opposite way, look again, repeat. The emerald ball in the viewport is doing exactly this, starting from the probe, and each dot on its trail is one step.
Worked example. At the gradient is . With a step size of the step is , which lands at . The height there is , down from . On this bowl every step multiplies both coordinates by , so the ball visits , then , and settles at the bottom after about 29 steps (the HUD's walker readout counts them).
Walk down the two hills from , , with step size .
Same landscape and step size, but start a tiny distance away, at , .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 7.1
06Step size: too small crawls, too big overshoots
The gradient only describes the ground right under your feet. A tiny step trusts it for a short distance and makes slow progress. A huge step trusts it far beyond where it holds, and can carry you clean over the valley and up the other side.
Worked example. On the round bowl the gradient is , so one step turns into , and the same for . Everything depends on that factor . The viewport is set to : the factor is , each step only 4% closer, and after 80 steps the ball is still short of the bottom with .
Set the step size to .
Now overshoot: set .
Push it past the limit: set .
Machine Learning Crash Course, Google, Gradient descent; Learning rate
Questions learners ask
- What is the difference between a derivative and a partial derivative?
- An ordinary derivative is the slope of a function with one input. A partial derivative is the slope of a function with several inputs when you change only one of them and hold the others fixed. Geometrically you slice the landscape along one direction and take the slope of the curve you get, so each partial derivative is an ordinary derivative of one slice.
- Why does the gradient point in the direction of steepest ascent?
- The slope you feel walking in a direction u of length 1 is the dot product of the gradient with u, which equals the length of the gradient times the cosine of the angle between them. The cosine is largest, 1, when the angle is zero, so the slope is largest when you walk along the gradient. It is zero when you walk at a right angle to it, which is the direction of the contour line.
- What is the learning rate in gradient descent?
- It is the number that multiplies the gradient to set the size of each step, often written as eta. Too small and training crawls; too large and each step overshoots the valley, so the error bounces around or grows. Practitioners usually try a few values spread over several powers of ten and keep the largest one whose error still falls smoothly.
Sources
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy
- Machine Learning Crash Course, Google