0.4 Derivatives: slopes, tangents and the chain rule
A derivative is the steepness of a curve at one exact spot, found by shrinking a gap until a slanted line between two points becomes the tangent.
01Steepness between two points
Walk along the cyan curve in the viewport. It is gentle near and steep further right. To put a number on that steepness, pick two points on the curve and join them with a straight line, the amber secant. Its slope tells you how fast the curve climbs, on average, between the two points.
Worked example. The emerald point is and the amber point is , because and . Going from to you move 1 across (the run) and up (the rise). Slope is rise divided by run: . The HUD shows the same number as secant slope.
Halve the gap: set , so moves to .
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Calculus 1: secant lines and average rate of change
02Shrink the gap: the secant becomes the tangent
The secant measures steepness between two points, but we want the steepness at alone. The trick is to slide toward . As the gap shrinks, the amber secant swings round until it lies on top of the emerald tangent.
Worked example. The viewport now has , so . Rise , run , slope . With the slope is , and with it is . The numbers close in on , the slope of the tangent at . Try it in the table: press Shrink h ÷ 10 and watch the emerald digits settle.
Press play on the whole story: shrinks from to , then the view zooms in on .
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 5.1, Differentiation of univariate functions
03Zoom in until the curve looks straight
Here is another way to see the same idea. Any smooth curve, viewed under a strong enough magnifying glass, looks like a straight line. The slope of that line is the derivative. The viewport shows with at , where .
Zoom in 100 times around : set the zoom slider to .
Worked example. Because the curve is almost straight near , you can predict nearby heights with the tangent: start at the height of and add slope times step. For at (height , slope ), a step of predicts . The true value is , very close. The smaller the step, the better the prediction.
Try a different curve: the sigmoid at , zoomed about ().
04The derivative is a new function
So far we measured the slope at one point. Now let travel along the curve and record the slope at every stop. Plot each slope as a height in the lower panel and you get a second curve: the slope graph . In the viewport sweeps along and the emerald trace below is drawn as it goes.
Worked example. At the sine curve climbs at slope . At the top of the hill, , it is flat: slope . At it falls at slope . Those three points, , and , lie on the emerald curve below, which is .
Pause at the top of the sine hill: , which is .
Sweep the cubic instead.
05Shortcut rules: powers, constants and sums
Shrinking works for any curve, but for common functions the limit has already been worked out once and for all. A handful of rules covers most of what you will meet. First, see why the slope of is :
- 1
Rise over run for f(x) = x², at any x.
Worked example. For : the power rule turns into ; the term has slope ; the sum rule adds them, so . At , where the viewport is, . The HUD agrees: the secant with has slope , and the true derivative is .
Find the bottom of the cubic's dip: move to .
A negative slope: the parabola at .
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Calculus 1: power rule and basic differentiation rules
06The chain rule: nudges multiply
Most functions in ML are chains: one step feeds the next. Think of linked gears. Turning turns , and turning turns . The viewport shows three number lines stacked top to bottom. A small nudge of (cyan arrow) moves (amber arrow), which moves (emerald arrow).
Worked example. Take and , at . Then and . The inner gear has slope : moves twice as far as . The outer gear has slope at . Nudge by : moves , and moves from to , about . Altogether moved about times as far as .
Swap the outer function for , keeping and .
Build a one-input neuron: and , at .
Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd: the chain rule in backpropagation
Deep Learning, Goodfellow, Bengio & Courville, Sec. 6.5.2, Chain rule of calculus
Questions learners ask
- What is a derivative in simple terms?
- A derivative is how steep a curve is at one exact point: how much the output changes for a tiny change in the input. You find it by drawing a straight line through two nearby points on the curve and moving them closer together; the slope of that line settles on the derivative. For y = x squared, the derivative at x = 1 is 2, so near that point the output grows about twice as fast as the input.
- Why does machine learning need derivatives?
- Training a model means adjusting its numbers to make its error smaller. The derivative of the error with respect to each number says which way to nudge it and how strongly the error will respond. Gradient descent repeats that nudge thousands of times, and backpropagation is the efficient way to compute all of those derivatives at once using the chain rule.
- How do you calculate a derivative numerically in Python?
- Evaluate the function a small step h on each side of the point and divide the difference by 2h: (f(x + h) - f(x - h)) / (2 * h), with h around 1e-5. This central difference is much more accurate than the one-sided version, and with NumPy the same line works on a whole array of x values at once. Libraries such as PyTorch compute exact derivatives automatically, and numerical derivatives are a common way to check them.
Sources
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Deep Learning, Goodfellow, Bengio & Courville
- Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy