2.2 Linear and logistic regression from first principles
Least squares drops the targets perpendicularly onto the span of the features, and logistic regression bends the same linear score through a sigmoid and fits it with cross-entropy, a convex loss whose gradient has the same shape.
01Least squares fits a plane
Start with one feature. The model is a straight line: is its height at (the intercept) and is how much rises for every unit of (the slope). The residual of a point is its vertical gap to the line, : positive above the line, negative below. A good line makes all the gaps small, and least squares measures "small" by the average of the squared gaps.
Linear regression with more features is the same idea: a weighted sum of features plus an intercept. With two features the model is a plane over the floor, and fitting it means choosing three numbers :
θ has shape (3,); y, ŷ and r have shape (40,). Leave out the column of ones and the plane is forced through the origin.
The 40 cyan points come from , with noise of standard deviation . The residual of point is its vertical gap to the plane, , drawn as a rose stick. Least squares minimises the mean of their squares:
The emerald plane starts flat at , where the MSE is , and gradient descent with step size tilts it into place in 22 steps. It stops at with MSE : close to the true , and close to the noise variance , which is the best any plane can do on average. Drag the noise slider and the points, the sticks and the fit move together.
Set the noise to : the targets become exactly .
Now push the noise to , more than four times the default.
Scoring a regression. The MSE is in squared units, so reports usually quote its square root, the RMSE, in the units of . The MAE averages instead, so one large error counts linearly rather than quadratically. $R^2$ compares the model's MSE with that of the constant prediction : , the share of the variance the model explains. On the viewport's 40 points at noise : RMSE , MAE , .
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 3.1 and 3.2
Machine Learning Crash Course, Google, Linear regression
02The normal equations
Gradient descent is not needed here. The MSE is a quadratic bowl in , so its minimum is where the gradient vanishes, and setting the gradient to zero gives a linear system with a closed-form solution. Do it first with one feature, where every step is ordinary calculus.
- 1
The bowl on the right of the widget above.
Both conditions say the same thing: the residuals are uncorrelated with each column of inputs (the constant 1 and ). With features there is one such condition per column, and writing them all at once in matrix form gives the normal equations.
- 1
Expand the squared norm. θᵀXᵀy and yᵀXθ are the same scalar, so they combine.
Read the second line again: at the optimum, . Every column of is orthogonal to the residual vector. For the column of ones that says , so a least-squares model with an intercept always has residuals that average zero. Hover to watch shrink to zero as the plane converges.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.2.1
Hands-On Machine Learning, Aurélien Géron, Ch. 4, The Normal Equation
03When XᵀX is singular
exists only if no column of is a linear combination of the others. The correlation slider now mixes toward , with , and the cyan points collapse onto the diagonal of the floor. A plane resting on a line of points can rotate about that line with almost no change in fit.
Two things go wrong. The coefficients are pinned down only along the data: OLS now returns , far from the true and although the noise is unchanged, and only the sum is well determined. And gradient descent crawls along the flat direction of the bowl: after its 150-step budget it sits at , a visibly different plane from the emerald outline, with MSE against the optimum's .
Back off to a milder correlation, , and restart the fit.
Now .
The everyday cause: one-hot encoding. A categorical feature such as colour {red, green, blue} becomes three 0/1 columns, one per value. In every row exactly one of them is 1, so the three columns add up to the column of ones: exact collinearity, , and is singular. This is the dummy-variable trap. Drop one dummy column (that category becomes the baseline the others are compared with), or drop the intercept, or regularise.
An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 3.3.3, Collinearity
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.2
The rest of this lesson is in Pro
- 04Least squares is a projection
- 05From regression to classification: log-odds and the sigmoid
- 06Cross-entropy: convex, with a familiar gradient
- 07Separability, regularisation and reading coefficients
- Python challenge: Least squares and logistic regression from scratch
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between linear and logistic regression?
- Linear regression predicts a real number as a weighted sum of features and is fitted by minimising squared error, which has a closed-form solution. Logistic regression predicts the probability of a class by passing the same weighted sum through a sigmoid, and is fitted by minimising cross-entropy, which is convex but needs an iterative solver. Both have a linear score; logistic regression's decision boundary, where the probability is one half, is a straight line or hyperplane.
- Why does logistic regression use cross-entropy instead of mean squared error?
- Cross-entropy is the negative log-likelihood of a Bernoulli model, and combined with the sigmoid it is convex in the weights with the simple gradient X transpose times (p minus y) over n. Mean squared error on sigmoid outputs is not convex, and its gradient becomes tiny when the model is confidently wrong, so training stalls exactly where it should correct the most.
- When should I use the normal equation instead of gradient descent?
- The normal equation solves least squares exactly in one step and needs no learning rate or feature scaling, but its cost grows with the cube of the number of features and it needs the whole dataset in memory. Use it, through a QR or SVD solver, for up to a few thousand features. Use gradient descent or SGD when there are very many features or rows, when data arrives in batches, or when the loss has no closed form, as in logistic regression.
Next in Classical Machine Learning
2.3Bias, variance and regularisationWhy more capacity can hurt, how ridge and lasso trade bias for variance, and how to choose λ with cross-validation.Sources
- An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
- Machine Learning Crash Course, Google
- Hands-On Machine Learning, Aurélien Géron
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Introduction to Linear Algebra, Gilbert Strang