2.3 Bias, variance and regularisation
Expected test error splits into bias², variance and irreducible noise, and a penalty on the coefficients buys a large cut in variance for a small rise in bias, with λ chosen by cross-validation rather than by the training error.
01Expected test error has three parts
A model is fitted to one training set, but we care about its error on new data, and a different training set would have given a different model. The viewport makes that concrete. The truth is , the white curve. Twenty inputs sit at , and each of 30 training sets draws fresh labels with noise of standard deviation . Every training set gets its own cubic fit , which enters at the front of the stack with its 20 cyan points and pushes the earlier fits back.
The intuition first, before any formula. Train the same model on 30 slightly different datasets. If the 30 answers disagree with each other, that is variance. If they agree with each other but are all wrong in the same way, that is bias. A dartboard shows both at once: each dart is one trained model, the bullseye is the truth. Try the four presets, then turn on the averaging toggle.
Once all 30 are drawn, the stack collapses onto the front plane and the fits contract onto their average , the bold emerald curve, leaving the amber gap to the truth. The spread you saw before the contraction is variance; the gap that remains is bias.
One input, real numbers. The amber-and-rose marker on the dashed vertical line is a probe at . The emerald beads are where each of the 30 cubic fits crosses that line: their values run from to . Their mean is (emerald dot) and the truth is (white dot). So at this input the squared bias is , the amber segment, and the variance, the average squared distance of the 30 values from , is , the rose bracket of standard deviation (). Drag Probe input x₀ in the cockpit; the HUD reads both numbers at every .
- 1
Add and subtract f(x) and the average fit. ε is the new label's noise: mean 0, variance σ², independent of the training set.
One row per fitted model. f̄ is the column mean; bias² averages (f̄ − f)² over the columns, variance averages each column's variance over the rows.
The inset at the bottom right splits the bar exactly this way. At degree 3 it reads , bias² and variance , so the expected test MSE is . The cyan tick marks the average training MSE, .
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 2.2.2, The Bias-Variance Trade-Off
The Elements of Statistical Learning, Hastie, Tibshirani & Friedman, Ch. 7.3
02Capacity: underfitting and overfitting
The degree sets the model's capacity: the fit uses plus an intercept, so it has parameters. The lower chart in the inset plots the four quantities against on a log scale: test MSE in white, training MSE in cyan, bias² in amber and variance in rose. The emerald line marks the current degree.
Underfitting. At a straight line cannot follow the sine: bias² dominates, variance is only , and training and test MSE nearly agree at and . Overfitting. The viewport is now at : 16 parameters for 20 points. The fits chase the noise of their own training set, the stack becomes a tangle of curves that leave the plot near the edges, and variance reaches while bias² is only . Test MSE is , more than twelve times its value at , while training MSE falls to .
Drop to a straight line, , and move the probe to the right edge, .
Walk up the degree: , no penalty.
Keep climbing: .
Drag from 1 to 15 and read the chart: bias² falls quickly and then flattens, variance grows slowly and then explodes, and their sum with is lowest at (0.1138), with a hair behind at 0.1140.
At , put the probe between the last two training inputs, .
Footnote: double descent. The slider stops at , below , the classical regime where this U-shape holds. When the number of parameters exceeds , least squares has infinitely many interpolating solutions, and the minimum-norm one, which gradient descent from zero converges to, can see test error fall again as capacity keeps growing (Belkin et al., 2019). The peak sits at the interpolation threshold, where the fit is forced through every noisy point; a well-tuned ridge penalty largely removes it.
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 2.2.2 and 7.1
Hands-On Machine Learning, Aurélien Géron, Ch. 4, Polynomial Regression
Machine Learning Crash Course, Google, Overfitting
03Ridge regression: penalise the squared norm
Instead of lowering the degree, keep all 15 powers and make large coefficients expensive. Ridge regression adds times the squared norm of the weights to the residual sum of squares. The powers are standardised first, so the penalty treats every coefficient alike, and the intercept is left out of it:
- 1
Setting the derivative in b to zero gives b̂. Substituting it centres every column and the targets: X_c = X − 1x̄ᵀ, y_c = y − ȳ. Centring removes the intercept from the problem, so the penalty cannot touch it.
The viewport is back at , now with . The tangle calms into a tight bundle: variance drops from to , bias² is , and the expected test MSE falls from to , close to the best low-degree fit. The training MSE rises a little, from to : the fits no longer chase every point.
Ridge spends its flexibility only where the data support it. Hover to read the effective degrees of freedom , defined in the next step: 11.84 at , 6.20 at and 4.69 at , for a model that nominally has 15 slopes. Push to 10 and the penalty wins: bias² climbs to and test MSE to . Too much regularisation is underfitting by another name.
Raise the penalty to () and put the probe at , where the unpenalised fits scattered most.
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 6.2.1, Ridge Regression
The Elements of Statistical Learning, Hastie, Tibshirani & Friedman, Ch. 3.4.1
Hands-On Machine Learning, Aurélien Géron, Ch. 4, Ridge Regression
The rest of this lesson is in Pro
- 04Shrinkage along singular directions
- 05Lasso: exact zeros from the L1 corner
- 06Choosing λ with k-fold cross-validation
- 07Regularisation in practice
- Python challenge: Ridge, soft thresholding and leak-free cross-validation
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the bias-variance tradeoff?
- The expected squared error of a model on new data splits into three parts: the squared bias, how far the average model over many training sets is from the truth; the variance, how much the model changes from one training set to another; and the irreducible noise in the labels. Flexible models have low bias but high variance, rigid models the reverse, and the best test error sits in between. Regularisation and more data reduce variance; more capacity or better features reduce bias.
- What is the difference between ridge and lasso regression?
- Both add a penalty on the size of the coefficients to the squared-error loss. Ridge penalises the sum of squared coefficients, has a closed-form solution and shrinks every coefficient smoothly without making any of them exactly zero. Lasso penalises the sum of absolute values, has no closed form in general, and sets some coefficients exactly to zero, so it also performs feature selection. With strongly correlated features the lasso picks one somewhat arbitrarily, which the elastic net, a mix of both penalties, fixes.
- How do I choose the regularization parameter lambda?
- Use k-fold cross-validation on the training data: for each value on a logarithmic grid, fit on k minus 1 folds, measure the error on the held-out fold, and average. Pick the value with the lowest average, or the largest value within one standard error of it for a simpler model. Fit every preprocessing step, such as scaling, inside each fold, and keep a separate test set to report the final error.
Next in Classical Machine Learning
2.4Decision trees, random forests and gradient boostingAxis-aligned partitions of feature space, bagging to cut variance, and boosting as gradient descent in function space.Sources
- An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
- Hands-On Machine Learning, Aurélien Géron
- Deep Learning, Goodfellow, Bengio & Courville
- The Elements of Statistical Learning, Hastie, Tibshirani & Friedman
- Machine Learning Crash Course, Google