Skip to content
LearnMLByDoing
Free

2.7 PCA and dimensionality reduction

32 minProintermediateHOMLMMLStrangCompendium

PCA finds the orthogonal directions along which centred data varies most, keeps the top k of them, and describes every sample with k coordinates whose reconstruction error is exactly the variance it throws away.

01Why reduce dimension

Feature vectors are long: a 28 × 28 image has 784 pixels, a tabular model can have hundreds of columns, a text embedding 768 or more coordinates. Three costs grow with the dimension dd. Sparsity: the average distance between two random points in the unit hypercube is 0.66 in 3D and grows like d/6\sqrt{d/6}, to roughly 408 in a million dimensions, so a fixed number of samples covers the space ever more thinly and distances concentrate. Storage and compute scale with dd. And people can look at two or three coordinates at most.

"Distances concentrate" is easiest to believe when you watch it. The widget scatters 200 random points in the unit cube [0,1]d[0, 1]^d and histograms all 19,900 distances between them. At d=3d = 3 the farthest point from point 1 is 8.38.3 times as far as its nearest neighbour. Slide dd up to 1000 and the histogram collapses into a spike: that ratio falls to 1.111.11, and the closest and the farthest of all 19,900 pairs differ by only 16%. When every point is about equally far from every other, "nearest" stops meaning much, which hurts k-nearest neighbours, clustering and vector search.

Loading figure
Distances are divided by d/6\sqrt{d/6}, the typical distance in dd dimensions, so every dd fits one axis. The raw mean distance is in the readouts: 0.660.66 at d=3d = 3, 12.912.9 at d=1000d = 1000.
X (data)[,]1,200 values

The lab's cloud: 400 samples in the rows, 3 features x₁, x₂, x₃ in the columns. Real data has the same layout with a much larger d.

Most data does not fill all dd directions evenly. The cloud in the viewport has three features, yet 95.4%95.4\% of its variance lies in one plane through its mean. Every few seconds the points slide perpendicularly onto that emerald plane and back. While they rest on it, each point is described by 2 numbers instead of 3, and the short rose segments show what that compression throws away.

X∈Rn×d  ⟶  Z∈Rn×k,k≪d\tok{data}{X}\in\mathbb{R}^{n\times d}\;\longrightarrow\;\tok{projection}{Z}\in\mathbb{R}^{n\times \tok{num_components}{k}},\qquad \tok{num_components}{k}\ll d
Dimensionality reduction replaces d coordinates per sample with k.

Hands-On Machine Learning, Aurélien Géron, Ch. 8, The curse of dimensionality; Main approaches

02The direction of maximum variance

Centre the data by subtracting the mean μ\tok{mean}{\mu}, the white sphere; hover μ\mu to see how far it sits from the origin. Projecting a centred point x~=x−μ\tilde{\mathbf{x}}=\mathbf{x}-\mu onto a unit vector u\mathbf{u} gives the scalar u⊤x~\mathbf{u}^{\top}\tilde{\mathbf{x}}, the dot product from lesson 0.2: how far the point reaches along u\mathbf{u}. For example x~=(2,1)\tilde{\mathbf{x}} = (2, 1) and u=(0.8,0.6)\mathbf{u} = (0.8, 0.6) give 0.8⋅2+0.6⋅1=2.20.8 \cdot 2 + 0.6 \cdot 1 = 2.2, so the point's foot on the line through u\mathbf{u} is 2.2 u=(1.76,1.32)2.2\,\mathbf{u} = (1.76, 1.32). PCA picks the u\mathbf{u} along which these scalars spread out most, because a direction with a large spread keeps the samples distinguishable after the projection.

Find that direction by hand first. The widget is a flat cousin of the cloud: 80 centred points in 2D. Drag the amber tip to turn u\mathbf{u}. Each emerald foot is a projected point and each rose segment is what the projection throws away. At the starting angle, 115°115°, the feet bunch up near the centre: the variance along u\mathbf{u} is only 0.2500.250, and the residuals carry 0.9660.966. Turn u\mathbf{u} until the emerald curve peaks, or press snap to u₁.

Loading figure
At u1\mathbf{u}_1 (about 36°36°) the kept variance is 0.9920.992 and the residual variance 0.2240.224, its minimum. The two always add up to about 1.221.22, the total variance tr⁡S\operatorname{tr}S: whatever the projection keeps, the residuals lose.
S=1n−1∑i=1n(xi−μ)(xi−μ)⊤\tok{covariance}{S}=\frac{1}{n-1}\sum_{i=1}^{n}(\mathbf{x}_i-\tok{mean}{\mu})(\mathbf{x}_i-\tok{mean}{\mu})^{\top}
Sample covariance, d × d. Hover S to draw its 2-standard-deviation ellipsoid.

Why the answer is an eigenvector: we maximise u⊤Su\mathbf{u}^{\top}S\mathbf{u} subject to ∥u∥=1\|\mathbf{u}\| = 1. The constraint enters through a Lagrange multiplier (lesson 1.4), and the multiplier λ\lambda turns out to be the eigenvalue of SS for that direction.

Maximum variance is an eigenvalue problem1/4
  1. 1
    Var⁡(u⊤x)=1n−1∑i(u⊤x~i)2=u⊤S u\operatorname{Var}(\mathbf{u}^{\top}\mathbf{x})=\frac{1}{n-1}\sum_{i}(\mathbf{u}^{\top}\tilde{\mathbf{x}}_i)^2=\mathbf{u}^{\top}S\,\mathbf{u}

    The variance of the projected data is a quadratic form in u.

SS is symmetric and positive semi-definite, so its eigenvectors are orthonormal and its eigenvalues are real and non-negative. For the cloud, λ=(0.967, 0.278, 0.061)\lambda=(0.967,\ 0.278,\ 0.061) and u1=(0.760, 0.566, 0.320)\mathbf{u}_1=(0.760,\ 0.566,\ 0.320). The viewport keeps only k=1k=1: the points rest on the emerald line through μ\mu along u1\mathbf{u}_1, which carries 0.967/1.305=74.1%0.967/1.305=74.1\% of the total variance. Each arrow is 2λi2\sqrt{\lambda_i} long, two standard deviations along its direction.

Keep two components instead of one.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 10.2, Maximum variance perspective

03PCA through the SVD

Forming SS and calling an eigensolver works, but libraries compute PCA from the singular value decomposition of the centred data matrix XcX_c, whose rows are the x~i⊤\tilde{\mathbf{x}}_i^{\top}. The SVD never forms Xc⊤XcX_c^{\top}X_c, which would square the condition number (the ratio of the largest to the smallest singular value, a measure of how much rounding errors get amplified) and lose precision, and it has truncated and randomised variants for large matrices.

Xc=U Σ V⊤,S=Xc⊤Xcn−1=V Σ2n−1 V⊤X_c=U\,\Sigma\,V^{\top},\qquad \tok{covariance}{S}=\frac{X_c^{\top}X_c}{n-1}=V\,\frac{\Sigma^{2}}{n-1}\,V^{\top}
U and V have orthonormal columns; Σ is diagonal with σ₁ ≥ σ₂ ≥ … ≥ 0.

The second identity is an eigendecomposition of SS, so the columns of VV are the principal directions and λi=σi2/(n−1)\tok{eigenvalue}{\lambda_i}=\tok{singular_value}{\sigma_i}^2/(n-1). For the cloud, σ=(19.64, 10.53, 4.91)\sigma=(19.64,\ 10.53,\ 4.91) and 19.642/399=0.96719.64^2/399=0.967. The projected coordinates, or scores, are Z=XcVk=UkΣk\tok{projection}{Z}=X_cV_k=U_k\Sigma_k, and the reconstruction is X^=ZVk⊤+1μ⊤\hat X=ZV_k^{\top}+\mathbf{1}\mu^{\top}.

X_c = U Σ Vᵀ (thin SVD)[,]1,200 values

U is 400 × 3, Σ and Vᵀ are 3 × 3. Centre first: the SVD of the raw X finds the direction of the mean, not of the spread.

V_kᵀ (components)[,]6 values

The first k rows of Vᵀ: orthonormal principal directions as rows, the components array in the challenge.

Z (scores)[,]800 values

Each sample's coordinates in the kept subspace: (X − μ) times the transposed components.

In the viewport k=2k=2 and the points rest on the plane spanned by u1\mathbf{u}_1 and u2\mathbf{u}_2; their positions in that plane are the rows of ZZ. Hover z\mathbf{z} to read the scores of the white point, or σ\sigma to convert singular values into variances.

Introduction to Linear Algebra, Gilbert Strang, Ch. 7.3, Principal component analysis by the SVD

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 10.4

Maths, CS & AI Compendium, Henry Ndubuaku, Matrix decompositions

The rest of this lesson is in Pro

  • 04Choosing k: explained variance and reconstruction error
  • 05Scale before you project
  • 06Where a linear projection fails
  • 07Manifold learning: unrolling the sheet
  • Python challenge: PCA from the SVD
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between PCA and SVD?
The singular value decomposition is a matrix factorisation that exists for any matrix. PCA is a statistical procedure that centres the data and then finds the directions of maximum variance. The two meet because the right singular vectors of the centred data matrix are the principal directions, and each eigenvalue of the covariance matrix equals the squared singular value divided by n minus 1. Most libraries compute PCA with an SVD for numerical stability.
How many principal components should I keep?
Look at the cumulative explained variance ratio and pick the smallest k that reaches a target such as 95 percent, or look for an elbow in the scree plot. When PCA feeds a model, treat k as a hyperparameter and choose it by validation score. The reconstruction error for a given k equals the sum of the eigenvalues you discard, which makes the trade-off explicit.
Do I need to standardise data before PCA?
Usually yes when features use different units or scales, because PCA maximises variance and a feature measured in larger units dominates the first component. Standardising makes PCA work on the correlation matrix. Fit the scaler and the PCA on training data only and apply them unchanged to validation, test and production data. Skip standardisation only when all features share one meaningful unit.

Next in Classical Machine Learning

2.8Precision, recall, ROC and calibrationSlide a decision threshold through two score distributions and watch the confusion matrix, ROC and precision-recall curves respond.

Sources

  • Hands-On Machine Learning, Aurélien Géron
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • Introduction to Linear Algebra, Gilbert Strang
  • Maths, CS & AI Compendium, Henry Ndubuaku