3.1 Backpropagation and non-linear decision boundaries
A hidden layer bends the input plane until two interlocked spirals can be split by a flat plane, and backpropagation is how the network learns that bend.
01Why a straight line cannot do it
The viewport shows two interlocked spirals: 150 emerald points (class 0) and 150 amber points (class 1), each arm winding about 1.1 turns around the origin. Every point is drawn at the height of its label, amber on top and emerald below. The translucent sheet is the model's current estimate of , and training pulls it up to the amber points and down to the emerald ones. The cyan curve is where that estimate crosses 0.5: the decision boundary.
Start with the simplest classifier, logistic regression. It scores a point with one affine function and squashes the score into a probability. One point with friendly numbers: , weights and bias give the score , and : a 69% vote for amber. In general:
The boundary is the zero set of an affine function, so it is a straight line. Whatever and training finds, each side of that line contains whole arcs of both spirals, and accuracy stays far from 100%. More epochs or a better optimiser cannot fix this. The limit is the hypothesis class, not the search.
The viewport is running the network of this lesson with its activation switched off (Activation: none). It has 16 hidden units, but each one is a linear function of , and a linear function of linear functions is still linear. So the whole model is one affine score, the cyan boundary is a straight line through the spirals, and the acc readout stalls at about 54% (chance is 50%).
Turn the non-linearity on: switch the activation to tanh and keep everything else.
Back to no activation, but with four times as many hidden units: .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.1
03In hidden space, a flat plane is enough
The output layer is logistic regression on . With two classes, the softmax depends only on the difference of the two logits:
Its decision boundary is a hyperplane in hidden space. The scene's vertical axis measures , the signed distance to that hyperplane, stretched so the margin is easy to see. At fold 1 the hyperplane is the flat cyan plane at height 0, and every correctly classified point lies on its own colour's side: amber above, emerald below.
Now pull the fold back to 0 and follow the cyan contour. It is the same set, , drawn in input coordinates. Because is non-linear, a flat cut in pulls back to a curved boundary in . Training changes both: reshapes the sheet and tilts the plane.
Make the spirals noisy: raise the noise to 0.25, so the two arms overlap.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 15
04Backpropagation: the chain rule, run backwards
Gradient descent (lesson 2.1) moves every weight a little against its gradient : how much the loss changes per unit nudge of that weight. The network in the viewport has weights and biases at , so it needs 82 gradients per update. Backpropagation gets all of them in one backward sweep. Before the matrix version, watch it on a network small enough to show every number.
Two things to notice in the widget. Every gradient is a product of local factors along the path from the weight to the loss, and the backward pass computes each shared factor once (here and ) and reuses it for every weight below it. When a number feeds two paths, as does, its gradient is the sum over the paths. With ReLU, a hidden unit whose passes a local factor of exactly 0, so every weight feeding it gets zero gradient.
- 1
Cross-entropy for a label y in {0, 1} and a predicted probability p of class 1, from the score s.
Training the full network minimises the mean cross-entropy over the training points:
Backpropagation computes every gradient in one backward sweep, reusing the values stored during the forward pass. Stack the batch into matrices: is , is , holds the predicted probabilities and the one-hot labels, and are the hidden pre-activations.
- 1
The result derived above, for every row at once: predicted minus true distribution. The 1/N comes from averaging the loss over the batch.
same shape as h
Every gradient has the shape of the parameter it updates. Counting the work: the forward pass costs one matrix product per layer (, then ), and the backward pass costs two per layer, one for the weight gradient () and one to send the error down (). That is why a training step takes roughly three times the arithmetic of evaluating the network. Hover to flash the points with the largest error : they sit where the sheet has not yet reached their label, and they dominate the next update.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.5
Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, micrograd
05Capacity, dead units and gradient checks
The scene is now set to with ReLU, with the three units' lines drawn in amber. Three ReLU units crease the plane along only three lines, so the boundary is piecewise linear with a handful of pieces, while the spirals need many more. Accuracy plateaus near 70% however long it trains. Drag Fold to 1: with the folded view shows all of hidden space, rotated and stretched vertically, and nothing is projected away.
Give the network more creases: ReLU units.
ReLU adds a second failure mode. If unit has a negative pre-activation for every input, then everywhere, its column of is zero and its weights never change again. This is a dead unit: in the 2-2-1 widget above it is a hidden unit whose activation factor reads 0. With , losing one unit removes a third of the model's capacity. Wider layers, smooth activations such as tanh and GELU, Leaky ReLU (a small slope instead of 0 for negative inputs) and careful initialisation make dead units rare.
Start every unit with a negative bias, , so many of them begin switched off on the whole dataset (, ReLU).
Before you trust a hand-written backward pass, compare it with central finite differences on a tiny network. The central difference has error , so with in float64 a relative error below means the backward pass is correct; the challenge harness accepts anything below . Errors near usually mean a missing , a missing transpose or the wrong activation derivative.
Deep Learning, Goodfellow, Bengio & Courville, Ch. 6.3.1
Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy, makemore part 4: backprop ninja
Questions learners ask
- What does backpropagation actually compute?
- Backpropagation computes the gradient of the loss with respect to every weight in one backward sweep through the network. It applies the chain rule layer by layer, starting from the output error p - y, and reuses the activations stored during the forward pass, so the cost is about the same as a forward pass.
- Why does a neural network need a non-linear activation function?
- Without a non-linearity, stacked linear layers collapse into a single linear map, so the decision boundary stays a straight line or hyperplane. Activations such as ReLU, tanh and GELU let hidden layers bend the input space so that classes which are tangled in the input, such as two spirals, become linearly separable in the hidden representation.
- How do I check that my backward pass is correct?
- Run a gradient check: for each parameter, compare the analytic gradient with the central difference (L(theta + eps) - L(theta - eps)) / (2 eps) on a tiny network, with eps around 1e-5 in float64. A relative error below 1e-6 means the backward pass is correct, and the challenge harness accepts anything below 1e-5; errors near 1e-1 usually point to a missing 1/N, a missing transpose or the wrong activation derivative.
Next in Deep Learning and Autograd
3.2Build a scalar autograd engineA micrograd-style Value class with topological-sort backprop — the core of every deep learning framework in 120 lines.Sources
- Deep Learning, Goodfellow, Bengio & Courville
- Neural Networks: Zero to Hero (micrograd, makemore), Andrej Karpathy