Skip to content
LearnMLByDoing
Free

0.1 Functions and graphs: a model is a function with knobs

25 minFree lessonbeginner3B1BKhanMML

A function is a machine that turns an input number into an output number, and a machine learning model is a function whose knobs get turned until its graph runs close to the data.

01A function is a machine

A function is a machine with one fixed rule: a number goes in, the rule is applied, and one number comes out. In the viewport the cyan dot on the horizontal axis is the input. It slides from left to right, the cyan guide carries it to the green line, and the green marker on the vertical axis shows the output.

The rule in the viewport is multiply by 2, then add 1. Feed it 2: 2×2=42 \times 2 = 4, then 4+1=54 + 1 = 5. Feed it −1-1: 2×(−1)=−22 \times (-1) = -2, then −2+1=−1-2 + 1 = -1. The same input always gives the same output. That is the one promise every function makes.

Loading figure
Drag the input or the two knobs and press Run it again to watch the number pass through each stage. The arithmetic underneath lights up in the same order.
f(x)=2 x+1\tok{y_out}{f(x)} = 2\,\tok{x_in}{x} + 1
The rule of the function in the viewport, written as a formula.

Stop the sweep and feed the machine one input of your choice: pause at x=−1x = -1.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 1: Functions

02Pairs of numbers draw a graph

Every time you feed the machine an input, you get a pair: the input and its output. Read the pair as directions: go xx across, then f(x)f(x) up, and put a dot there. In the viewport, each time the sweep passes a whole number it leaves a green dot labelled with its pair.

For f(x)=2x+1f(x) = 2x + 1: f(−2)=−3f(-2) = -3, f(−1)=−1f(-1) = -1, f(0)=1f(0) = 1, f(1)=3f(1) = 3 and f(2)=5f(2) = 5. Those five pairs are the five green dots. Inputs in between fill the gaps: x=0.5x = 0.5 gives f(0.5)=2f(0.5) = 2, halfway between the dots at (0,1)(0, 1) and (1,3)(1, 3). Every input at once makes the bright line, called the graph of ff.

Loading figure
Collect a few pairs with Add (x, f(x)) to the table: each one becomes a dot. Then turn on Join the dots. Changing a knob starts a new table, because a new rule makes new pairs.

Pause the sweep with the input at x=1.5x = 1.5, between two of the dots.

03The straight line: slope and intercept

Every straight line has two knobs. The slope mm says how steep it is: how far up you go for each step to the right. The intercept bb says where the line crosses the vertical axis. The viewport is set to m=0.5m = 0.5 and b=1.5b = 1.5. The amber triangle goes 1 across and 0.5 up, and the amber ring sits at height 1.5 on the vertical axis.

f(x)=m x+b\tok{y_out}{f(x)} = \tok{slope}{m}\,\tok{x_in}{x} + \tok{intercept}{b}
Every straight line (except a vertical one) has this form.

To find the slope from two points, count. From (1,2)(1, 2) to (3,3)(3, 3) you move 2 across, the run, and 1 up, the rise. The slope is rise over run: 1/2=0.51 / 2 = 0.5. The intercept then follows from either point:

Finding b from one point on the line1/3
  1. 1
    2=0.5×1+b2 = 0.5 \times 1 + b

    The line passes through (1, 2): put x = 1 and f(x) = 2 into f(x) = 0.5x + b.

Loading figure
Drag either point; they snap to halves. The viewport's line follows, so you see the same mm and bb in 3D. Put both points at the same xx and the slope breaks.

Make the line fall: set m=−1m = -1 and keep b=1.5b = 1.5.

Khan Academy: Algebra, Calculus and Statistics, Khan Academy, Algebra 1: Linear equations and graphs

04Other curve families and their knobs

A straight line is one family of functions. Change the kind of rule and you get a different family, each with its own knobs. Switch families with the Family control and the curve morphs from one shape to the next.

Quadratic. The new knob aa multiplies x2x^2 and bends the line into a U. The viewport has a=0.5a = 0.5, m=0m = 0 and b=0b = 0, so f(2)=0.5×4=2f(2) = 0.5 \times 4 = 2 and f(−2)=2f(-2) = 2 as well: squaring forgets the sign, which is why the U is symmetric. A negative aa turns the U upside down.

f(x)=a x2+m x+b\tok{y_out}{f(x)} = \tok{curvature}{a}\,\tok{x_in}{x}^2 + \tok{slope}{m}\,\tok{x_in}{x} + \tok{intercept}{b}
Quadratic: the line m·x + b plus a bend a·x².

Exponential, f(x)=b emxf(x) = b\,e^{mx}: each step of 1 to the right multiplies the output by the same factor, eme^{m}, so it grows faster and faster when m>0m > 0 and fades toward 0 when m<0m < 0. Sigmoid, f(x)=σ(mx+b)f(x) = \sigma(mx + b): an S-shaped curve squeezed between 0 and 1, the two dashed lines. Here mm sets how steep the S is and bb slides it sideways. The same knob names do different jobs in different families.

Flip and shift the U: set a=−0.5a = -0.5, m=1m = 1 and b=3b = 3.

Switch to the exponential with b=1b = 1 and m=0.5m = 0.5.

Switch to the sigmoid with m=3m = 3 and b=−3b = -3.

05Data, gaps and the total squared error

Now the job turns around. Instead of a rule you are given data: twelve measured pairs, the cyan dots. You want a line that passes close to all of them. For each dot, the rose stick is the vertical gap between the dot and the line, how far the line's output misses the real output at that input.

The viewport starts with a flat line, m=0m = 0 and b=1.5b = 1.5. The dot at x=2.3x = 2.3 has y=3.1y = 3.1, but the line outputs 1.51.5, so the gap is 3.1−1.5=1.63.1 - 1.5 = 1.6. Squaring gives 1.62=2.561.6^2 = 2.56, the area of the rose square drawn on that stick. While the scene plays it visits the dots one by one and adds up their squares. The total is E=15.91E = 15.91.

E=∑i=112(yi−f(xi))2\tok{sse}{E} = \sum_{i=1}^{12} \big(\tok{residual}{y_i - f(x_i)}\big)^2
Total squared error: square the gap of every data point, then add them up.

Improve the line by hand: set m=0.5m = 0.5 and keep b=1.5b = 1.5.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.1 and 9.2

06Fitting: turn the knobs to shrink the error

Fitting means turning the knobs until the error is as small as it can get. You did it by hand in the last step. With Auto-fit on, the scene does it for you: many times a second it nudges mm and bb a little in whichever direction makes EE smaller, and you can watch the line swing into place while the squares melt.

It starts from a bad guess, m=−0.5m = -0.5 and b=3b = 3, where E=84.96E = 84.96. Within about two seconds the slope is close to right; then the line slides down and settles at m=0.50m = 0.50, b=1.53b = 1.53 and E=1.23E = 1.23. No straight line scores lower on these twelve points. Your hand-tuned line from the last step was close, at 1.241.24.

mnew=mold−η×(how fast E changes as m turns)\tok{slope}{m}_{\text{new}} = \tok{slope}{m}_{\text{old}} - \eta \times \big(\text{how fast } \tok{sse}{E} \text{ changes as } \tok{slope}{m} \text{ turns}\big)
One nudge. The same rule turns b (and a, for the quadratic) at the same time.

Pick the wrong family: switch to the sigmoid, starting from m=0m = 0 and b=1.5b = 1.5, with auto-fit on.

Give the model a spare knob: switch to the quadratic with a=0.5a = 0.5, m=0m = 0 and b=1.5b = 1.5, auto-fit on.

Mathematics for Machine Learning, Deisenroth, Faisal & Ong, Ch. 9.2

Compiling scene
Loading editor

Questions learners ask

What is a function in maths, in simple terms?
A function is a rule that takes an input number and gives back exactly one output number. You can picture it as a machine: put 2 into the rule 'multiply by 2, then add 1' and 5 comes out, every time. Written as a formula this is f(x) = 2x + 1, and f(2) = 5 means the output for the input 2. Plotting every input against its output draws the function's graph.
What do the slope and the intercept mean in y = mx + b?
The slope m is how steep the line is: how far it goes up for each step of 1 to the right, found from any two points as rise divided by run. A negative slope means the line goes down. The intercept b is the height where the line crosses the vertical axis, which is the output when x is 0. Changing b moves the whole line up or down; changing m tilts it around that crossing point.
Why is a machine learning model called a function with parameters?
A model takes an input, such as a picture or a sentence or a list of measurements, and returns an output, such as a label or a number, so it is a function. Its rule contains adjustable numbers called parameters, like the slope and intercept of a line. Training means turning those numbers until the model's outputs are close to the right answers on example data, usually by making a total or average squared error as small as possible.

Next in Maths Primer

0.2Vectors: arrows, adding, scaling and the dot productDrag arrows around a plane, add and scale them, measure their length, and discover the dot product as a score of how much two directions agree.

Sources

  • Essence of Linear Algebra, 3Blue1Brown (Grant Sanderson)
  • Khan Academy: Algebra, Calculus and Statistics, Khan Academy
  • Mathematics for Machine Learning, Deisenroth, Faisal & Ong