Skip to content
LearnMLByDoing
Free

2.8 Precision, recall, ROC and calibration

32 minPro, with free previewintermediateHOMLISLRMLCCFawcett

A classifier only ranks cases by score; every metric in this lesson comes from where you cut that ranking, so slide one threshold through two score distributions and watch the confusion matrix, the ROC and precision-recall curves, and calibration respond.

01Scores, a threshold and four outcomes

A binary classifier does not output labels. It outputs a score ss per case, higher meaning more likely positive, and a separate decision rule turns the score into a label. Every metric in this lesson is a property of that rule, not of the model alone.

y^={1s≥t0s<t\hat y=\begin{cases}1 & s\ge \tok{threshold}{t}\\ 0 & s<\tok{threshold}{t}\end{cases}
The translucent plane is the threshold. Cases to its right are predicted positive.

Start with cases you can count. Below are 20 cases, 6 positives on the top row and 14 negatives on the bottom row, each placed at its score. Drag the cyan threshold line. Every dot right of it is predicted positive, and each dot takes the colour of its cell: TP emerald, FN amber, FP rose, TN grey.

Loading figure
At t=0.5t=0.5: 5 TP, 1 FN, 3 FP and 11 TN, so precision 5/8=0.6255/8=0.625, recall 5/6=0.8335/6=0.833 and accuracy 16/20=0.8016/20=0.80. Now switch on the 60 easy negatives: accuracy rises to 71/80=0.88871/80=0.888 while precision falls to 5/13=0.3855/13=0.385, because 5 of the new negatives score above 0.50.5. The threshold is shared with the 3D plane.

Now imagine 1000 cases instead of 20. The dots pile up so densely that each row becomes a smooth ribbon, which is what the viewport draws. A latent score is Gaussian within each class, z∣y=1∼N(+d/2,1)z\mid y{=}1\sim\mathcal N(+\tok{separation}{d}/2,1) and z∣y=0∼N(−d/2,1)z\mid y{=}0\sim\mathcal N(-\tok{separation}{d}/2,1), and the model reports s=σ(z)s=\sigma(z). Here d=1.5d=1.5 and the prevalence is π=0.2\tok{prevalence}{\pi}=0.2. The front ribbon is the positive class and the back ribbon the negative class; each ribbon's height is its density times its class count, which is why the positives are a quarter as tall.

C (confusion matrix)[,]4 values

Rows are the true class, columns the decision. The four cells sum to n, and every metric below is a ratio of cells.

The plane cuts each ribbon in two. Positives to its right are TP\tok{tp}{\mathrm{TP}} (emerald), positives to its left are FN\tok{fn}{\mathrm{FN}} (amber), negatives to its right are FP\tok{fp}{\mathrm{FP}} (rose) and negatives to its left are true negatives (grey). The area of a ribbon piece is the number of cases in it, the continuous version of counting dots. At t=0.5t=0.5 the four areas are, per 1000 cases, TP 155, FN 45, FP 181 and TN 619. The bars on the floor are those counts, laid out like the ribbons: actual positives in front, predicted positives on the right.

Keep t=0.5t=0.5 and double the separation to d=3d=3.

Hands-On Machine Learning, Aurélien Géron, Ch. 3, Confusion matrix

An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani, Sec. 4.4.2

02Precision, recall, F1 and specificity

The metrics differ in what they condition on. Recall and the false positive rate divide by the true class, so each looks at one ribbon. Precision divides by the decision, so it looks at everything right of the plane, both ribbons at once.

TPR=TPTP+FN,FPR=FPFP+TN\tok{recall}{\mathrm{TPR}}=\frac{\tok{tp}{\mathrm{TP}}}{\tok{tp}{\mathrm{TP}}+\tok{fn}{\mathrm{FN}}},\qquad \tok{fpr}{\mathrm{FPR}}=\frac{\tok{fp}{\mathrm{FP}}}{\tok{fp}{\mathrm{FP}}+\mathrm{TN}}
Each divides by a true class: recall (sensitivity) within the positives, FPR within the negatives.
Prec=TPTP+FP,spec=TNTN+FP\tok{precision}{\mathrm{Prec}}=\frac{\tok{tp}{\mathrm{TP}}}{\tok{tp}{\mathrm{TP}}+\tok{fp}{\mathrm{FP}}},\qquad \text{spec}=\frac{\mathrm{TN}}{\mathrm{TN}+\tok{fp}{\mathrm{FP}}}
Precision divides by the decision. Specificity, the negatives' recall, equals 1 − FPR.
Precision from the rates, and F1 as a harmonic mean1/4
  1. 1
    P(y=1∣y^=1)=P(y^=1∣y=1) P(y=1)P(y^=1)P(y{=}1\mid \hat y{=}1)=\frac{P(\hat y{=}1\mid y{=}1)\,P(y{=}1)}{P(\hat y{=}1)}

    Precision is the probability that an alarm is a real positive; Bayes' rule rewrites it. P(ŷ=1 | y=1), the chance a positive is flagged, is recall written as a probability.

At t=0.6t=0.6 the HUD reads recall 0.6350.635, FPR 0.1240.124 (specificity 0.8760.876), precision 0.5610.561, F1 0.5960.596 and accuracy 0.8280.828. Hover Prec\mathrm{Prec} to see which regions enter it.

Move the threshold down from 0.60.6 to t=0.4t=0.4.

Machine Learning Crash Course, Google, Classification: accuracy, recall, precision and related metrics

Hands-On Machine Learning, Aurélien Géron, Ch. 3, Precision and recall

03The ROC curve, and AUC as a ranking probability

Each threshold gives one point (FPR,TPR)(\mathrm{FPR},\mathrm{TPR}). Sweeping tt from 1 down to 0 traces the ROC curve from (0,0)(0,0), where nothing is flagged, to (1,1)(1,1), where everything is. Watch the viewport: the plane sweeps the score axis from 0 to 1 and the emerald operating point rides the curve from (1,1)(1,1) back to (0,0)(0,0), highlighting the part it has traced, then returns to t=0.5t=0.5.

AUC=∫01TPR  d FPR\tok{auc}{\mathrm{AUC}}=\int_0^1\tok{recall}{\mathrm{TPR}}\;d\,\tok{fpr}{\mathrm{FPR}}
The shaded area in the panel. The dashed diagonal is a random ranking, AUC = 0.5.

Before the integral, a small example. Pick one positive and one negative at random and ask: does the positive get the higher score? With three positives and three negatives there are nine pairs. Count a win as 1, a tie as 12\tfrac12 and a loss as 0, and divide by 9. That fraction is the AUC.

Loading figure
The positives win 6 pairs, lose 2 and tie 1 (p3p_3 and n2n_2 both score 0.400.40), so AUC =6.5/9=0.722=6.5/9=0.722. Click a cell to draw its pair. Drag n1n_1 down to 0.350.35: both losses become wins and the AUC rises to 8.5/9=0.9448.5/9=0.944; only the tie at 0.400.40 keeps it below 1.
Why AUC is the probability that a positive outranks a negative1/5
  1. 1
    TPR(t)=P(s+≥t),FPR(t)=P(s−≥t)\mathrm{TPR}(t)=P(s^{+}\ge t),\qquad \mathrm{FPR}(t)=P(s^{-}\ge t)

    s⁺ and s⁻ are the scores of a random positive and a random negative. With f⁻ the negative score density, d FPR = −f⁻(t) dt.

Hover AUC\mathrm{AUC}: the scene draws random positive-negative pairs and links them, emerald when the positive scores higher. The running win rate settles near 0.8560.856. Push dd to 33 and the AUC becomes 0.9830.983.

Set the separation to d=0d=0: positives and negatives get the same score distribution.

An Introduction to ROC Analysis, Tom Fawcett, 2006, Sec. 3, 5 and 7

Machine Learning Crash Course, Google, Classification: ROC and AUC

The rest of this lesson is in Pro

  • 04Class imbalance: when accuracy and ROC mislead
  • 05Choosing a threshold from costs
  • 06Calibration and the Brier score
  • 07More than two classes, and metrics that survive imbalance
  • Python challenge: Confusion counts, ROC and AUC from scratch
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between precision and recall?
Recall is the share of actual positives the model catches: true positives divided by all positives. Precision is the share of the model's positive predictions that are correct: true positives divided by all predicted positives. Lowering the threshold usually raises recall and lowers precision. Recall does not depend on how common positives are, while precision falls as positives become rarer.
When should I use a precision-recall curve instead of an ROC curve?
Use the precision-recall curve when the positive class is rare and false alarms matter, as in fraud detection, information retrieval or rare-disease screening. The ROC curve plots rates that are each computed within one class, so it looks the same at 1% and 50% prevalence and can look excellent while most alarms are false. The PR curve's baseline is the prevalence, so compare its area to that, not to 0.5.
How do I choose a classification threshold?
Pick it from the cost of each error on a validation set. With a false-positive cost and a false-negative cost, choose the threshold that minimises expected cost, which is where the ROC curve's slope equals (1 - prevalence) times the false-positive cost divided by prevalence times the false-negative cost. If the scores are calibrated probabilities, that is the threshold cost_FP / (cost_FP + cost_FN). Without explicit costs, fix a minimum recall or precision required by the application, or maximise an F-beta score with the beta that reflects the trade-off.

Sources

  • Hands-On Machine Learning, Aurélien Géron
  • An Introduction to Statistical Learning, James, Witten, Hastie & Tibshirani
  • Machine Learning Crash Course, Google
  • An Introduction to ROC Analysis, Tom Fawcett, 2006