2.7 PCA and dimensionality reduction
PCA finds the orthogonal directions along which centred data varies most, keeps the top k of them, and describes every sample with k coordinates whose reconstruction error is exactly the variance it throws away.
01Why reduce dimension
Feature vectors are long: a 28 × 28 image has 784 pixels, a tabular model can have hundreds of columns, a text embedding 768 or more coordinates. Three costs grow with the dimension . Sparsity: the average distance between two random points in the unit hypercube is 0.66 in 3D and grows like , to roughly 408 in a million dimensions, so a fixed number of samples covers the space ever more thinly and distances concentrate. Storage and compute scale with . And people can look at two or three coordinates at most.
"Distances concentrate" is easiest to believe when you watch it. The widget scatters 200 random points in the unit cube and histograms all 19,900 distances between them. At the farthest point from point 1 is times as far as its nearest neighbour. Slide up to 1000 and the histogram collapses into a spike: that ratio falls to , and the closest and the farthest of all 19,900 pairs differ by only 16%. When every point is about equally far from every other, "nearest" stops meaning much, which hurts k-nearest neighbours, clustering and vector search.
The lab's cloud: 400 samples in the rows, 3 features x₁, x₂, x₃ in the columns. Real data has the same layout with a much larger d.
Most data does not fill all directions evenly. The cloud in the viewport has three features, yet of its variance lies in one plane through its mean. Every few seconds the points slide perpendicularly onto that emerald plane and back. While they rest on it, each point is described by 2 numbers instead of 3, and the short rose segments show what that compression throws away.
Hands-On Machine Learning, Aurélien Géron, Ch. 8, The curse of dimensionality; Main approaches
02The direction of maximum variance
Centre the data by subtracting the mean , the white sphere; hover to see how far it sits from the origin. Projecting a centred point onto a unit vector gives the scalar , the dot product from lesson 0.2: how far the point reaches along . For example and give , so the point's foot on the line through is . PCA picks the along which these scalars spread out most, because a direction with a large spread keeps the samples distinguishable after the projection.
Find that direction by hand first. The widget is a flat cousin of the cloud: 80 centred points in 2D. Drag the amber tip to turn . Each emerald foot is a projected point and each rose segment is what the projection throws away. At the starting angle, , the feet bunch up near the centre: the variance along is only , and the residuals carry . Turn until the emerald curve peaks, or press snap to u₁.
Why the answer is an eigenvector: we maximise subject to . The constraint enters through a Lagrange multiplier (lesson 1.4), and the multiplier turns out to be the eigenvalue of for that direction.
- 1
The variance of the projected data is a quadratic form in u.
is symmetric and positive semi-definite, so its eigenvectors are orthonormal and its eigenvalues are real and non-negative. For the cloud, and . The viewport keeps only : the points rest on the emerald line through along , which carries of the total variance. Each arrow is long, two standard deviations along its direction.
Keep two components instead of one.
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 10.2, Maximum variance perspective
03PCA through the SVD
Forming and calling an eigensolver works, but libraries compute PCA from the singular value decomposition of the centred data matrix , whose rows are the . The SVD never forms , which would square the condition number (the ratio of the largest to the smallest singular value, a measure of how much rounding errors get amplified) and lose precision, and it has truncated and randomised variants for large matrices.
The second identity is an eigendecomposition of , so the columns of are the principal directions and . For the cloud, and . The projected coordinates, or scores, are , and the reconstruction is .
U is 400 × 3, Σ and Vᵀ are 3 × 3. Centre first: the SVD of the raw X finds the direction of the mean, not of the spread.
The first k rows of Vᵀ: orthonormal principal directions as rows, the components array in the challenge.
Each sample's coordinates in the kept subspace: (X − μ) times the transposed components.
In the viewport and the points rest on the plane spanned by and ; their positions in that plane are the rows of . Hover to read the scores of the white point, or to convert singular values into variances.
Introduction to Linear Algebra, Gilbert Strang, Ch. 7.3, Principal component analysis by the SVD
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 10.4
Maths, CS & AI Compendium, Henry Ndubuaku, Matrix decompositions
The rest of this lesson is in Pro
- 04Choosing k: explained variance and reconstruction error
- 05Scale before you project
- 06Where a linear projection fails
- 07Manifold learning: unrolling the sheet
- Python challenge: PCA from the SVD
- Knowledge check, 4 questions
Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.
The Python challenge and knowledge check for this lesson are part of Pro.
Questions learners ask
- What is the difference between PCA and SVD?
- The singular value decomposition is a matrix factorisation that exists for any matrix. PCA is a statistical procedure that centres the data and then finds the directions of maximum variance. The two meet because the right singular vectors of the centred data matrix are the principal directions, and each eigenvalue of the covariance matrix equals the squared singular value divided by n minus 1. Most libraries compute PCA with an SVD for numerical stability.
- How many principal components should I keep?
- Look at the cumulative explained variance ratio and pick the smallest k that reaches a target such as 95 percent, or look for an elbow in the scree plot. When PCA feeds a model, treat k as a hyperparameter and choose it by validation score. The reconstruction error for a given k equals the sum of the eigenvalues you discard, which makes the trade-off explicit.
- Do I need to standardise data before PCA?
- Usually yes when features use different units or scales, because PCA maximises variance and a feature measured in larger units dominates the first component. Standardising makes PCA work on the correlation matrix. Fit the scaler and the PCA on training data only and apply them unchanged to validation, test and production data. Skip standardisation only when all features share one meaningful unit.
Next in Classical Machine Learning
2.8Precision, recall, ROC and calibrationSlide a decision threshold through two score distributions and watch the confusion matrix, ROC and precision-recall curves respond.Sources
- Hands-On Machine Learning, Aurélien Géron
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Introduction to Linear Algebra, Gilbert Strang
- Maths, CS & AI Compendium, Henry Ndubuaku