Skip to content
LearnMLByDoing
Free

1.3 Probability, Gaussians and maximum likelihood

32 minProintermediateMMLISLRCompendiumDLPML

A Gaussian is a mean and a covariance matrix: the covariance draws its ellipses, a Cholesky factor samples it, maximum likelihood recovers both from data, and slicing it at one coordinate produces a regression line, which is why least squares is Gaussian maximum likelihood.

01Random variables and densities

A random variable assigns a number to every outcome of an experiment. For a continuous variable, probabilities come from a density p(x)≥0p(x)\ge0: the probability of landing in a region is the volume under the density over that region, and the total volume is 1.

P(X∈A)=∫Ap(x) dx,∫Rdp(x) dx=1P(X\in A)=\int_A \tok{density}{p(x)}\,dx,\qquad \int_{\mathbb{R}^d}\tok{density}{p(x)}\,dx=1
A density is not a probability. It can exceed 1; only its integral over a region is a probability.

The Gaussian (normal) density with mean μ\mu and standard deviation σ\sigma is the bell in the widget below. The mean sets where the bell is centred; the standard deviation sets how wide it is. Drag the two cyan handles: the shaded area between them is the probability of landing there.

Loading figure
The standard normal, μ=0\mu=0 and σ=1\sigma=1. The interval [−1,1][-1,1] holds 68.3%68.3\% of the probability; drag bb to 22 and aa to −2-2 for 95.4%95.4\%. Now push σ\sigma below 0.40.4: the peak climbs past the dashed line at density 1, yet the total area stays 1.

Worked example. Adult heights in some population have mean 170170 cm and standard deviation 1010 cm. A height of 190190 cm is (190−170)/10=2(190-170)/10=2 standard deviations above the mean, and about 2.3%2.3\% of people are taller (half of the 4.6%4.6\% outside ±2σ\pm2\sigma). Subtracting the mean and dividing by σ\sigma maps any Gaussian onto the standard one, so a single table of N(0,1)\mathcal{N}(0,1) answers every question about every Gaussian.

N(x;μ,σ2)=1σ2πexp⁡ ⁣(−(x−μ)22σ2),z=x−μσ∼N(0,1)\mathcal{N}(x;\tok{mean}{\mu},\sigma^2)=\frac{1}{\sigma\sqrt{2\pi}}\exp\!\Big(-\frac{(x-\tok{mean}{\mu})^2}{2\sigma^2}\Big),\qquad z=\frac{x-\tok{mean}{\mu}}{\sigma}\sim\mathcal{N}(0,1)
Standardisation. The z-score is the distance from the mean in units of σ; 68%, 95% and 99.7% of the mass lies within 1, 2 and 3 σ.

The viewport draws a two-dimensional Gaussian as a surface over the (x1,x2)(x_1,x_2) floor, with mean μ=(0.4, −0.3)\mu=(0.4,\,-0.3), σ1=1.2\sigma_1=1.2, σ2=0.7\sigma_2=0.7 and correlation ρ=0\rho=0. With ρ=0\rho=0 the surface factorises into two 1D bells, p(x1,x2)=N(x1;0.4,1.22) N(x2;−0.3,0.72)p(x_1,x_2)=\mathcal{N}(x_1;0.4,1.2^2)\,\mathcal{N}(x_2;-0.3,0.7^2), and its peak is 1/(2π⋅1.2⋅0.7)=0.1891/(2\pi\cdot1.2\cdot0.7)=0.189. Cyan points rain down from the same distribution and pile up where the surface is high.

A point at x1=2.8x_1=2.8 has z1=(2.8−0.4)/1.2=2.0z_1=(2.8-0.4)/1.2=2.0: two standard deviations out along x1x_1. Standardising every feature this way, with the mean and standard deviation of the training split, puts features on one scale before gradient descent, k-nearest neighbours or regularisation compare them.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 6.2 and 6.5

Deep Learning, Goodfellow, Bengio & Courville, Ch. 3.3 and 3.9.3

02The multivariate Gaussian: covariance is shape

Two definitions first. The expectation E[x]\mathbb{E}[x] is the average value of xx over the distribution, its mean. The covariance Cov⁡(xi,xj)=E[(xi−μi)(xj−μj)]\operatorname{Cov}(x_i,x_j)=\mathbb{E}\big[(x_i-\mu_i)(x_j-\mu_j)\big] is positive when xix_i and xjx_j tend to be above their means together, negative when one tends to be above while the other is below, and Cov⁡(xi,xi)\operatorname{Cov}(x_i,x_i) is the variance σi2\sigma_i^2. Dividing by σiσj\sigma_i\sigma_j gives the correlation ρ∈[−1,1]\rho\in[-1,1].

In dd dimensions a Gaussian is set by a mean vector and a covariance matrix with entries Σij=Cov⁡(xi,xj)\Sigma_{ij}=\operatorname{Cov}(x_i,x_j). It is symmetric and positive definite: every direction has positive variance, w⊤Σw>0w^{\top}\Sigma w>0 for every w≠0w\neq0, because w⊤Σww^{\top}\Sigma w is the variance of the projection w⊤xw^{\top}x. In 2D it is Σ=(σ12ρσ1σ2ρσ1σ2σ22)\Sigma=\begin{pmatrix}\sigma_1^2&\rho\sigma_1\sigma_2\\\rho\sigma_1\sigma_2&\sigma_2^2\end{pmatrix}, the three numbers the cockpit exposes as σ1\sigma_1, σ2\sigma_2 and ρ\rho.

p(x)=1(2π)d/2 ∣Σ∣1/2exp⁡ ⁣(−12 dM2),dM2=(x−μ)⊤Σ−1(x−μ)\tok{density}{p(x)}=\frac{1}{(2\pi)^{d/2}\,|\tok{covariance}{\Sigma}|^{1/2}}\exp\!\Big(-\tfrac12\,\tok{mahalanobis}{d_M^2}\Big),\qquad \tok{mahalanobis}{d_M^2}=(x-\tok{mean}{\mu})^{\top}\tok{covariance}{\Sigma}^{-1}(x-\tok{mean}{\mu})
The density depends on x only through the Mahalanobis distance, so its level sets are the rings on the floor.
Why the contours are ellipses1/4
  1. 1
    Σ=UΛU⊤,Λ=diag⁡(λ1,λ2),U⊤U=I\Sigma=U\Lambda U^{\top},\qquad \Lambda=\operatorname{diag}(\lambda_1,\lambda_2),\quad U^{\top}U=I

    Σ is symmetric, so it has orthonormal eigenvectors uᵢ (the columns of U) and real eigenvalues; positive definite means every λᵢ > 0.

The viewport is at σ1=1.2\sigma_1=1.2, σ2=0.7\sigma_2=0.7, ρ=0.6\rho=0.6, so Σ=(1.440.5040.5040.49)\Sigma=\begin{pmatrix}1.44&0.504\\0.504&0.49\end{pmatrix} with eigenvalues λ1=1.658\tok{eigen}{\lambda_1}=1.658 and λ2=0.272\tok{eigen}{\lambda_2}=0.272. The major axis is tilted 23.3∘23.3^\circ above the x1x_1 axis with half-length λ1=1.29\sqrt{\lambda_1}=1.29; the minor half-length is 0.520.52. The peak rose from 0.1890.189 to 0.2370.237 because ∣Σ∣=0.452|\Sigma|=0.452 is 1−ρ2=0.641-\rho^2=0.64 times its uncorrelated value: the same unit of probability now sits on a narrower ridge.

Drag ρ\rho. The rings shear into a tilted ellipse and the arrows rotate, while the spread of each coordinate on its own stays σ1\sigma_1 and σ2\sigma_2. The back walls show this: the emerald curves are the marginal densities p(x1)=N(0.4, 1.22)p(x_1)=\mathcal{N}(0.4,\,1.2^2) and p(x2)=N(−0.3, 0.72)p(x_2)=\mathcal{N}(-0.3,\,0.7^2), the bell you get by ignoring the other coordinate, and the cyan bars are histograms of the landed samples' coordinates. The marginal variances set the bounding box; the correlation sets the shape inside it.

Flip the correlation to ρ=−0.8\rho=-0.8.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 4.4 and 6.5

Probabilistic Machine Learning: An Introduction, Kevin P. Murphy, Sec. 3.2.1 and 3.2.2

03Sampling with the Cholesky factor

A random number generator gives uniforms, and the Box-Muller transform turns two uniforms into two independent standard normals z∼N(0,I)z\sim\mathcal{N}(0,I). To give them a covariance, factor Σ=LL⊤\Sigma=LL^{\top} with LL lower-triangular, the Cholesky factor, and map every zz through it:

x=μ+Lz,Cov⁡(x)=L E[zz⊤] L⊤=LL⊤=Σx=\tok{mean}{\mu}+\tok{cholesky}{L}z,\qquad \operatorname{Cov}(x)=\tok{cholesky}{L}\,\mathbb{E}[zz^{\top}]\,\tok{cholesky}{L}^{\top}=\tok{cholesky}{L}\tok{cholesky}{L}^{\top}=\tok{covariance}{\Sigma}
Any square root of Σ would do; the Cholesky factor is the cheapest to compute and is triangular.
L=(σ10ρσ2σ21−ρ2)=(1.200.420.56)\tok{cholesky}{L}=\begin{pmatrix}\sigma_1&0\\\rho\sigma_2&\sigma_2\sqrt{1-\rho^2}\end{pmatrix}=\begin{pmatrix}1.2&0\\0.42&0.56\end{pmatrix}
The 2D factor in closed form, at the viewport's σ₁ = 1.2, σ₂ = 0.7, ρ = 0.6.

Read the rows: x1=μ1+1.2 z1x_1=\mu_1+1.2\,z_1 uses the first normal only, and x2=μ2+0.42 z1+0.56 z2x_2=\mu_2+0.42\,z_1+0.56\,z_2 borrows 0.42 z10.42\,z_1 from it. That shared z1z_1 is the correlation. The viewport rains 2,000 samples, each landing at μ+Lzi\mu+Lz_i for a fixed, seeded stream of ziz_i, so dragging σ\sigma or ρ\rho slides the whole cloud instead of redrawing it. Hover L\tok{cholesky}{L} to see its two columns as arrows.

Z, X (samples)[,]4,000 values

One sample per row. The whole batch is X = mu + Z @ L.T: an (n, d) by (d, d) product, no Python loop. μ broadcasts across rows.

The same factor evaluates the density. Solve Ly=x−μLy=x-\mu by forward substitution; then dM2=∥y∥2d_M^2=\|y\|^2 and log⁡∣Σ∣=2∑ilog⁡Lii\log|\Sigma|=2\sum_i\log L_{ii}, so neither Σ−1\Sigma^{-1} nor ∣Σ∣|\Sigma| is ever formed:

log⁡p(x)=−d2log⁡2π−∑ilog⁡Lii−12∥L−1(x−μ)∥2\log\tok{density}{p(x)}=-\tfrac d2\log2\pi-\sum_{i}\log\tok{cholesky}{L}_{ii}-\tfrac12\big\|\tok{cholesky}{L}^{-1}(x-\tok{mean}{\mu})\big\|^2
The form the challenge asks for, vectorised over the rows of X.

Undo the Cholesky map: slide Whiten to 1, so every sample moves from μ+Lzi\mu+Lz_i to μ+zi\mu+z_i.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 4.3 and 6.5.4

The rest of this lesson is in Pro

  • 04Maximum likelihood: fitting μ and Σ
  • 05Conditioning: slicing the density
  • 06Least squares is Gaussian maximum likelihood
  • 07Z-scores, Mahalanobis outliers and normality
  • Python challenge: Log-density, MLE and conditioning
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between Mahalanobis distance and Euclidean distance?
Euclidean distance treats every direction alike. Mahalanobis distance first rescales by the covariance matrix, so it measures distance in standard deviations along each principal axis and accounts for correlations between features. Points at the same Mahalanobis distance from the mean lie on the same density contour of a Gaussian, which makes it the natural score for multivariate outliers.
Why does the maximum likelihood estimate of variance divide by n instead of n minus 1?
Maximising the Gaussian likelihood over the mean and the variance gives the average squared deviation from the sample mean, which divides by n. Because the sample mean is fitted to the same data, that estimate is biased low by a factor of (n − 1)/n. Dividing by n − 1 gives the unbiased sample variance. The two differ noticeably only for small samples.
Why is least squares the same as maximum likelihood?
If each target equals a linear prediction plus independent Gaussian noise with constant variance, the log-likelihood of the data is a constant minus the sum of squared errors divided by twice the noise variance. Maximising it over the weights is therefore the same as minimising the sum of squared errors. Under other noise models the equivalence breaks: Laplace noise leads to absolute error instead.

Next in Mathematical Foundations

1.4Convexity, Lagrange multipliers and dualityWhen a minimum is guaranteed to be global, and how constraints reshape the landscape — the math behind SVMs and regularisation.

Sources

  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong
  • An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Deep Learning, Goodfellow, Bengio & Courville
  • Probabilistic Machine Learning: An Introduction, Kevin P. Murphy