Skip to content
LearnMLByDoing
Free

3.2 Build a scalar autograd engine

40 minProintermediateKarpathyDLAD surveyParr

Autograd records the arithmetic of the forward pass as a graph, then walks it backwards once to deliver the gradient of the loss with respect to every number that went in.

01Computation is a graph of Values

Training a model means nudging every weight a little in the direction that lowers the loss (gradient descent, lesson 2.1). For that we need ∂L/∂w\partial L/\partial w for every weight: how much the loss moves per unit nudge of that weight. An autograd engine computes all of them automatically from the code that computes LL. Here is how. Wrap every scalar in a Value object that stores four things: its number (data), the Values it was computed from (_prev), the operation that produced it (_op), and a slot for its gradient (grad, starting at 0). Running ordinary Python on such objects records the computation as a side effect. The expression below builds the graph in the viewport: 13 value nodes (the glass blocks) and 7 operation nodes (the spheres).

L=(tanh⁡(w1x1+w2x2+b)−y)2\tok{loss}{L}=\big(\tanh(w_1x_1+w_2x_2+b)-y\big)^2
A single tanh neuron with a squared error. Intermediate names: s = w₁x₁ + w₂x₂, n = s + b, t = tanh n, d = t − y.

The emerald pulses are the forward pass: each op runs once its inputs exist, and its block lights up with its value. The viewport is at x1=1.5x_1=1.5 with w1=0.5w_1=0.5, x2=−1x_2=-1, w2=0.25w_2=0.25, b=ln⁡2−12≈0.193b=\ln 2-\tfrac12\approx0.193 and target y=1y=1, which gives s=0.5s=0.5, n=ln⁡2n=\ln 2, t=0.6t=0.6, d=−0.4d=-0.4 and L=0.16\tok{loss}{L}=0.16. The middle line of each block is data; the bottom line is grad, still 0.000 until the amber backward pass returns from the right.

Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd: the Value object

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.1

02Local derivatives and the chain rule

Each operation only needs its local derivative: how its output moves when one input moves and the others are held fixed. Four rules cover every op in this lesson:

∂(a+b)∂a=1,∂(a b)∂a=b,∂ak∂a=k ak−1,∂tanh⁡a∂a=1−tanh⁡2a\tok{local}{\frac{\partial(a+b)}{\partial a}}=1,\qquad \frac{\partial(a\,b)}{\partial a}=b,\qquad \frac{\partial a^{k}}{\partial a}=k\,a^{k-1},\qquad \frac{\partial\tanh a}{\partial a}=1-\tanh^2 a
Subtraction is addition of −b, so ∂(a − b)/∂b = −1.

Write vˉ=∂L/∂v\tok{grad}{\bar v}=\partial L/\partial v for the gradient of the loss with respect to node vv, read v-bar.

The chain rule says that an op v=f(u,… )v=f(u,\dots) passes its own gradient back to each input uu, scaled by the local derivative:

uˉ  +=  ∂v∂u vˉ\tok{grad}{\bar u}\;\tok{accum}{\mathrel{+}=}\;\tok{local}{\frac{\partial v}{\partial u}}\,\tok{grad}{\bar v}
One line per input of every op. The += matters in step 4.
Backward through the neuron at x₁ = 1.51/5
  1. 1
    Lˉ=1\bar L=1

    The seed: the loss changes one-for-one with itself. backward() sets L.grad = 1 before anything else.

These are the numbers on the blocks once the amber pass has finished, and the bars on top show their size and sign: amber for positive, cyan for negative. Hover ∂v/∂u\tok{local}{\partial v/\partial u} to read the local derivatives of the op the animation is on.

Freeze the pass where every gradient is ready and scrub it yourself.

Check the meaning of wˉ1=−0.768\bar w_1 = -0.768 with a real nudge: move w1w_1 from 0.5 to 0.51 and LL goes from 0.16 to 0.1525, a change of −0.0075-0.0075. The gradient predicted −0.768×0.01=−0.0077-0.768 \times 0.01 = -0.0077. The small gap is curvature, which a straight-line prediction ignores.

Loading figure
Pick a leaf and a nudge size. The actual change in LL (rose) and the gradient's prediction (amber) agree better the smaller the nudge.

The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018, Sec. 4.5, the chain rules

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.2

03Reverse mode: one backward pass, every gradient

A node may only pass its gradient back once every node that uses it has added its contribution. A topological order π\tok{topo}{\pi} lists each node after all of its inputs, so walking it in reverse visits each node after all of its consumers. micrograd builds one with a depth-first search from LL that appends a node after all of its children. The small emerald number on each block is its position in that order: x1x_1 0, w1w_1 1, x1w1x_1w_1 2, x2x_2 3, w2w_2 4, x2w2x_2w_2 5, ss 6, bb 7, nn 8, tt 9, yy 10, dd 11, LL 12. This is one valid order; yours may differ, because your Value stores _prev as a set and may visit inputs in another order.

Loading figure
The depth-first search behind those numbers, one call at a time. Step until the list is complete, then keep stepping to walk it backwards. Switch to the fan-out graph to see a node reached twice but appended once.

Slow the animation to 0.3× with the indices on.

Lˉ=1;for v in reversed(π):uˉ+=∂v∂u vˉ  for each input u of v\tok{grad}{\bar L}=1;\qquad \text{for } v \text{ in reversed}(\tok{topo}{\pi}):\quad \tok{grad}{\bar u}\mathrel{+}=\tok{local}{\frac{\partial v}{\partial u}}\,\tok{grad}{\bar v}\ \ \text{for each input } u \text{ of } v
The whole of backward(). Each node's _backward closure runs exactly once.

Each op does a constant amount of extra work in the backward pass, so one reverse sweep costs a small multiple of the forward pass and yields ∂L/∂v\partial L/\partial v for every node at once. Forward mode differentiates the other way round: it carries v˙=∂v/∂xi\dot v=\partial v/\partial x_i for one chosen input alongside the values, so the full gradient needs one pass per input. That is 6 passes for the six leaves here, and PP passes for a network with PP parameters.

∇L (gradient over the leaves)[]6 values

x₁, w₁, x₂, w₂, b, y. Reverse mode fills the whole vector in one backward pass; forward mode fills one entry per pass.

Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018, Sec. 3.1 and 3.2

Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5.3 and 6.5.9

The rest of this lesson is in Pro

  • 04Fan-out: gradients accumulate
  • 05A neuron: the tanh derivative
  • 06From scalars to tensors: what PyTorch does
  • 07Training on your engine
  • and 1 more step
  • Python challenge: Build a scalar autograd engine
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is autograd in PyTorch?
Autograd is PyTorch's reverse-mode automatic differentiation engine. During the forward pass it records every operation on tensors that require gradients as a node in a graph. Calling backward on a scalar loss walks that graph in reverse topological order, applies each operation's local derivative with the chain rule, and adds the result into the .grad of every leaf tensor. The graph is rebuilt on every forward pass.
Why does PyTorch accumulate gradients instead of overwriting them?
A value used in several places affects the loss through several paths, and the chain rule adds those contributions, so the engine has to sum into .grad. The same mechanism lets you accumulate gradients over several micro-batches before one optimizer step. The cost is that you must reset gradients with optimizer.zero_grad() before each new backward pass, or every step uses the sum of all previous gradients.
What is the difference between forward-mode and reverse-mode automatic differentiation?
Forward mode carries the derivative with respect to one input alongside the values, so a full gradient needs one pass per input. Reverse mode runs the forward pass, then propagates the derivative of one output backwards to all inputs in a single sweep. Training has millions of inputs (parameters) and one scalar loss, so deep learning uses reverse mode, also called backpropagation; its price is storing the intermediate values of the forward pass.

Next in Deep Learning and Autograd

3.3Initialisation, normalisation and training stabilityWhy activations explode or vanish, and how Kaiming init, BatchNorm and LayerNorm keep signals well-conditioned.

Sources

  • Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy
  • Deep Learning, Goodfellow, Bengio & Courville
  • Automatic Differentiation in Machine Learning: a Survey, Baydin et al., 2018
  • The Matrix Calculus You Need For Deep Learning, Parr & Howard, 2018