Skip to content
LearnMLByDoing
Free

2.2 Linear and logistic regression from first principles

30 minProintermediateISLRMLCCHOMLMMLStrang

Least squares drops the targets perpendicularly onto the span of the features, and logistic regression bends the same linear score through a sigmoid and fits it with cross-entropy, a convex loss whose gradient has the same shape.

01Least squares fits a plane

Start with one feature. The model y^=θ0+θ1x\hat y = \theta_0 + \theta_1 x is a straight line: θ0\theta_0 is its height at x=0x = 0 (the intercept) and θ1\theta_1 is how much y^\hat y rises for every unit of xx (the slope). The residual of a point is its vertical gap to the line, r=y−y^r = y - \hat y: positive above the line, negative below. A good line makes all the gaps small, and least squares measures "small" by the average of the squared gaps.

Loading figure
Twelve points from y=1+0.8xy = 1 + 0.8x plus noise of standard deviation 0.350.35. Drag the two white handles (or the amber dot in the MSE bowl): each rose square has area ri2r_i^2, and the MSE is their average area. The flat line y^=0\hat y = 0 scores 1.581.58; least squares, (θ0,θ1)=(0.957, 0.759)(\theta_0, \theta_1) = (0.957,\ 0.759), scores 0.0930.093, the bottom of the bowl.

Linear regression with more features is the same idea: a weighted sum of features plus an intercept. With two features the model is a plane over the (x1,x2)(x_1, x_2) floor, and fitting it means choosing three numbers θ=(θ0,θ1,θ2)\theta = (\theta_0, \theta_1, \theta_2):

y^=θ0+θ1x1+θ2x2=X θ\tok{yhat}{\hat y}=\theta_0+\theta_1x_1+\theta_2x_2=\tok{design}{X}\,\tok{theta}{\theta}
Each row of X is one observation (1, x₁, x₂). The leading 1 carries the intercept.
X (design matrix)[,]120 values

θ has shape (3,); y, ŷ and r have shape (40,). Leave out the column of ones and the plane is forced through the origin.

The 40 cyan points come from y=1.0+0.8x1−0.6x2+εy = 1.0 + 0.8x_1 - 0.6x_2 + \varepsilon, with noise ε\varepsilon of standard deviation 0.350.35. The residual of point ii is its vertical gap to the plane, ri=yi−y^ir_i = y_i - \hat y_i, drawn as a rose stick. Least squares minimises the mean of their squares:

MSE(θ)=1n∑i=1nri2=1n∥y−Xθ∥2\tok{mse}{\mathrm{MSE}}(\theta)=\frac1n\sum_{i=1}^{n}\tok{residual}{r_i}^{2}=\frac1n\big\|\tok{target}{y}-\tok{design}{X}\tok{theta}{\theta}\big\|^{2}
Hover MSE to draw every squared residual as a square: the MSE is their average area.

The emerald plane starts flat at θ=0\theta = 0, where the MSE is 2.442.44, and gradient descent with step size 0.150.15 tilts it into place in 22 steps. It stops at θ^=(0.988, 0.766, −0.615)\hat\theta = (0.988,\ 0.766,\ -0.615) with MSE 0.1120.112: close to the true (1.0, 0.8, −0.6)(1.0,\ 0.8,\ -0.6), and close to the noise variance 0.352=0.12250.35^2 = 0.1225, which is the best any plane can do on average. Drag the noise slider and the points, the sticks and the fit move together.

Set the noise to 00: the targets become exactly y=1.0+0.8x1−0.6x2y = 1.0 + 0.8x_1 - 0.6x_2.

Now push the noise to 1.51.5, more than four times the default.

Scoring a regression. The MSE is in squared units, so reports usually quote its square root, the RMSE, in the units of yy. The MAE averages ∣ri∣|r_i| instead, so one large error counts linearly rather than quadratically. $R^2$ compares the model's MSE with that of the constant prediction yˉ\bar y: R2=1−MSE/Var(y)R^2 = 1 - \mathrm{MSE}/\mathrm{Var}(y), the share of the variance the model explains. On the viewport's 40 points at noise 0.350.35: RMSE 0.3340.334, MAE 0.2810.281, R2=0.918R^2 = 0.918.

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 3.1 and 3.2

Machine Learning Crash Course, Google, Linear regression

02The normal equations

Gradient descent is not needed here. The MSE is a quadratic bowl in θ\theta, so its minimum is where the gradient vanishes, and setting the gradient to zero gives a linear system with a closed-form solution. Do it first with one feature, where every step is ordinary calculus.

One feature: slope = cov / var1/3
  1. 1
    MSE(θ0,θ1)=1n∑i(yi−θ0−θ1xi)2\mathrm{MSE}(\theta_0,\theta_1)=\frac1n\sum_{i}\big(y_i-\theta_0-\theta_1x_i\big)^2

    The bowl on the right of the widget above.

Both conditions say the same thing: the residuals are uncorrelated with each column of inputs (the constant 1 and xx). With dd features there is one such condition per column, and writing them all at once in matrix form gives the normal equations.

From the MSE to the normal equations1/4
  1. 1
    MSE(θ)=1n(y−Xθ)⊤(y−Xθ)=1n(y⊤y−2 θ⊤X⊤y+θ⊤X⊤X θ)\mathrm{MSE}(\theta)=\tfrac1n(y-X\theta)^{\top}(y-X\theta)=\tfrac1n\big(y^{\top}y-2\,\theta^{\top}X^{\top}y+\theta^{\top}X^{\top}X\,\theta\big)

    Expand the squared norm. θᵀXᵀy and yᵀXθ are the same scalar, so they combine.

θ^=(X⊤X)−1X⊤y\tok{theta}{\hat\theta}=(\tok{design}{X}^{\top}\tok{design}{X})^{-1}\tok{design}{X}^{\top}\tok{target}{y}
In the viewport this gives θ̂ = (0.988, 0.766, −0.615), the thin emerald outline the moving plane settles into.

Read the second line again: at the optimum, X⊤r=0X^{\top}r = 0. Every column of XX is orthogonal to the residual vector. For the column of ones that says ∑iri=0\sum_i r_i = 0, so a least-squares model with an intercept always has residuals that average zero. Hover r\tok{residual}{r} to watch X⊤r/nX^{\top}r/n shrink to zero as the plane converges.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.2.1

Hands-On Machine Learning, Aurélien Géron, Ch. 4, The Normal Equation

03When XᵀX is singular

(X⊤X)−1(X^{\top}X)^{-1} exists only if no column of XX is a linear combination of the others. The correlation slider now mixes x2x_2 toward x1x_1, x2←ρ x1+1−ρ2 x2x_2 \leftarrow \rho\,x_1 + \sqrt{1-\rho^2}\,x_2 with ρ=0.998\rho = 0.998, and the cyan points collapse onto the diagonal of the floor. A plane resting on a line of points can rotate about that line with almost no change in fit.

κ(X⊤X)=λmax⁡λmin⁡=110.60.098≈1133\kappa\big(\tok{design}{X}^{\top}\tok{design}{X}\big)=\frac{\lambda_{\max}}{\lambda_{\min}}=\frac{110.6}{0.098}\approx1133
At ρ = 0 the same ratio is 1.41. Hover X to read it live while you drag the slider.

Two things go wrong. The coefficients are pinned down only along the data: OLS now returns θ^=(0.988, 1.005, −0.839)\hat\theta = (0.988,\ 1.005,\ -0.839), far from the true 0.80.8 and −0.6-0.6 although the noise is unchanged, and only the sum θ1+θ2≈0.17\theta_1 + \theta_2 \approx 0.17 is well determined. And gradient descent crawls along the flat direction of the bowl: after its 150-step budget it sits at (0.988, 0.178, −0.013)(0.988,\ 0.178,\ -0.013), a visibly different plane from the emerald outline, with MSE 0.1150.115 against the optimum's 0.1120.112.

Loading figure
The same 40 points seen from above the bowl: MSE over the two slopes, with θ0\theta_0 set to its best value at every point. At ρ=0.998\rho = 0.998 the contours are a long, thin valley along θ1+θ2≈0.17\theta_1 + \theta_2 \approx 0.17. The amber path is the viewport's gradient descent from θ=0\theta = 0: it drops into the valley and stalls at (0.178, −0.013)(0.178,\ -0.013), far from the emerald OLS point (1.005, −0.839)(1.005,\ -0.839). Drag the white probe along the valley floor: the MSE hardly changes. The ρ\rho slider here also moves the 3D plane.

Back off to a milder correlation, ρ=0.9\rho = 0.9, and restart the fit.

Now ρ=0.99\rho = 0.99.

The everyday cause: one-hot encoding. A categorical feature such as colour ∈\in {red, green, blue} becomes three 0/1 columns, one per value. In every row exactly one of them is 1, so the three columns add up to the column of ones: exact collinearity, ρ=1\rho = 1, and X⊤XX^{\top}X is singular. This is the dummy-variable trap. Drop one dummy column (that category becomes the baseline the others are compared with), or drop the intercept, or regularise.

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Ch. 3.3.3, Collinearity

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.2

The rest of this lesson is in Pro

  • 04Least squares is a projection
  • 05From regression to classification: log-odds and the sigmoid
  • 06Cross-entropy: convex, with a familiar gradient
  • 07Separability, regularisation and reading coefficients
  • Python challenge: Least squares and logistic regression from scratch
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between linear and logistic regression?
Linear regression predicts a real number as a weighted sum of features and is fitted by minimising squared error, which has a closed-form solution. Logistic regression predicts the probability of a class by passing the same weighted sum through a sigmoid, and is fitted by minimising cross-entropy, which is convex but needs an iterative solver. Both have a linear score; logistic regression's decision boundary, where the probability is one half, is a straight line or hyperplane.
Why does logistic regression use cross-entropy instead of mean squared error?
Cross-entropy is the negative log-likelihood of a Bernoulli model, and combined with the sigmoid it is convex in the weights with the simple gradient X transpose times (p minus y) over n. Mean squared error on sigmoid outputs is not convex, and its gradient becomes tiny when the model is confidently wrong, so training stalls exactly where it should correct the most.
When should I use the normal equation instead of gradient descent?
The normal equation solves least squares exactly in one step and needs no learning rate or feature scaling, but its cost grows with the cube of the number of features and it needs the whole dataset in memory. Use it, through a QR or SVD solver, for up to a few thousand features. Use gradient descent or SGD when there are very many features or rows, when data arrives in batches, or when the loss has no closed form, as in logistic regression.

Next in Classical Machine Learning

2.3Bias, variance and regularisationWhy more capacity can hurt, how ridge and lasso trade bias for variance, and how to choose λ with cross-validation.

Sources

  • An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
  • Machine Learning Crash Course, Google
  • Hands-On Machine Learning, Aurélien Géron
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Introduction to Linear Algebra, Gilbert Strang