0.3 Matrices: tables of numbers that move space
A 2×2 matrix is four numbers that tell every point of the plane where to move, and once you can watch that movement, multiplying matrices and vectors becomes a picture you can check by eye.
01A matrix is a table that moves the plane
Look at the viewport. The faint square grid is the plane before anything happens. The brighter grid is the same plane after a matrix has moved it. While the animation plays, the plane glides from its starting position to the moved one and back. Notice what never changes: grid lines stay straight, they stay evenly spaced, and the origin stays where it is.
A matrix is written as a small table of numbers. The one in the viewport is this:
Here is the key idea of the whole lesson: read the table by its columns. Column 1, , is where the emerald arrow lands; starts as one step along . Column 2, , is where the amber arrow lands; it starts as one step up . In the viewport stays where it was, and tips over to , taking every vertical grid line with it.
Why do two arrows decide everything? Every point is an amount of plus an amount of : the cyan point is two steps of and one of . The matrix keeps those amounts and swaps in the moved arrows, so lands on . Know where the two arrows go, and you know where every point goes. The F inside the emerald unit square is carried along in the same way.
Swap the shear for a quarter turn: column 1 becomes and column 2 becomes .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3, Linear transformations and matrices
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2 and 2.7
02Matrix times vector: a weighted sum of columns
The viewport now uses a new matrix. The faint cyan arrow is the vector and the bright cyan arrow is where the matrix sends it, written . The two thin arrows show the route: one step along the emerald column, then two steps along the amber column.
- 1
Column picture: the entries of v say how much of each column to take, 1 of column 1 and 2 of column 2.
Change the probe to : one step along column 1, then one step backwards along column 2.
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3
Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Linear algebra: matrix-vector products
03Stretch, rotate, shear and flip
A few basic moves cover most of what a 2×2 matrix can do, and the preset menu in the cockpit has one of each. The viewport starts on a stretch: column 1 is , so doubles in length, while stays at . The grid is pulled sideways, the unit square becomes a 2-by-1 rectangle, and the probe lands on .
You can predict every picture from the columns alone. In the quarter turn, goes to and goes to : both arrows turn by 90 degrees. In the mirror, goes to while stays, so left and right swap over.
Apply the mirror: lands on and stays at .
Freeze a shear halfway: pause the animation at progress .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 3
Introduction to Linear Algebra, Gilbert Strang, Sec. 8.1
04The determinant: how much area grows
Watch the emerald unit square. It starts with area 1, and the matrix turns it into a slanted box called a parallelogram. In the viewport its sides are the two columns, and : a base of 2 and a height of 1.5, so its area is . Every grid cell grows by the same factor of 3, because the moved grid stays evenly spaced.
The sign carries information too. A negative determinant means the plane has been flipped over: the F reads backwards and the square turns rose. A determinant of zero means the square has been flattened onto a line, and points that started in different places now land on top of each other.
Flip it: keep column 1 at and send below it, to .
Squash it: send to , exactly half of column 1, which is now .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 6, The determinant
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 4.1
05Matrix times matrix: one move after another
Do a quarter turn, then a shear. The result is again one move of the plane that keeps lines straight and the origin fixed, so it is again a matrix. The viewport shows that combined matrix: watch head to and to .
- 1
The columns of R: where the turn sends the two arrows.
Do the same two moves in the other order, shear first and then turn: .
Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson), Ch. 4, Matrix multiplication as composition
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2.1
06A neural network layer: matrix times input, plus a bias
A layer of a neural network takes a list of input numbers and produces a list of output numbers. Its main ingredient is the move you now know: multiply the input by a weight matrix . Then it adds a bias vector , which slides the result. In the viewport the matrix is the one from step 2, and the bias shifts the grid, the square and the F as one piece; the grey arrow shows where the origin slides to.
Here 2 inputs and 2 outputs. A layer that reads a 28×28 image (784 inputs) into 128 outputs has a 128 × 784 weight matrix: 100,352 numbers, each one a knob that training adjusts.
Once a bias is added the origin moves, so on its own is a matrix move followed by a slide; grid lines still stay straight and evenly spaced. Real networks then pass every output through a small bending function, called an activation, so that a stack of layers can do far more than one matrix could.
Move the bias to .
Deep Learning, Goodfellow, Bengio & Courville, Ch. 2.1 and 2.2
Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Sec. 2.2
Questions learners ask
- What is a matrix, and why does machine learning use so many of them?
- A matrix is a rectangular table of numbers. Geometrically, a 2 by 2 matrix moves every point of the plane: its first column is where the arrow (1, 0) lands and its second column is where (0, 1) lands. Machine learning uses matrices because a layer of a neural network is a matrix multiplication plus a bias, and because a whole dataset can be stored as a matrix with one row per example, so a single multiplication processes all of it at once.
- How do you multiply a matrix by a vector?
- Take each entry of the vector as an amount of the matching column, then add the scaled columns: with columns (2, 1) and (-1, 1) and the vector (1, 2), the answer is 1 times (2, 1) plus 2 times (-1, 1), which is (0, 3). The row method gives the same numbers: each output entry is one row of the matrix multiplied entry by entry with the vector and added up. In NumPy you write A @ v.
- Why does the order of matrix multiplication matter?
- Because each matrix is a move of space, and the product AB means do B first and then A. Turning a shape and then shearing it usually gives a different result from shearing first and then turning, so AB and BA are usually different matrices. Their determinants always match, though, because the area factors multiply in either order.
Sources
- Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
- Mathematics for Machine Learning, Deisenroth, Faisal & Ong
- Khan Academy: Algebra, Calculus and Statistics, Khan Academy
- Deep Learning, Goodfellow, Bengio & Courville
- Introduction to Linear Algebra, Gilbert Strang