Skip to content
LearnMLByDoing
Free

6.1 Quantisation and efficient inference

32 minProintermediateGCP-PMLEMLA-C02CompendiumD2L

Quantisation stores each weight as a small integer code plus a shared scale, which cuts the bytes every generated token has to stream from memory at the price of a rounding error you can predict, measure and contain.

01Why quantise: decoding is bandwidth bound

Start with a kitchen picture. A chef who can chop ten onions a second is no faster than the corridor that brings onions from the pantry. A GPU is in the same position: it can multiply far faster than it can fetch the numbers to multiply. Memory bandwidth is the width of that corridor, the bytes per second that can travel from the GPU's memory to its compute units (about 1 TB/s here).

Generating one token with a decoder-only LLM at batch size 1 multiplies the current hidden state by every weight matrix once. That is about 2 FLOPs per parameter (one multiply, one add), but every parameter also has to travel from GPU memory (HBM) to the compute units. A 7B model in FP16 holds 14 GB of weights; at 1 TB/s of bandwidth reading them takes 14 ms, a ceiling of about 71 tokens per second however many FLOPs the chip has. Decoding small batches is memory-bandwidth bound, so bytes per parameter set the speed.

A floating-point number stores a sign, an exponent (the range: how large or small it can get) and a mantissa (the precision: how many significant digits it keeps). Bytes per parameter: FP32 4, FP16 and BF16 2 (BF16 keeps FP32's 8 exponent bits, so the same range, and drops mantissa bits), FP8 1, INT8 1, INT4 0.5. For 7×1097\times10^9 parameters that is 28, 14, 7, 7 and 3.5 GB of weights, before the KV cache and activations.

tokens/s  ≲  memory bandwidthbytes of weights=1000 GB/s7×109⋅b/8 bytes\text{tokens/s}\;\lesssim\;\frac{\text{memory bandwidth}}{\text{bytes of weights}}=\frac{1000\ \text{GB/s}}{7\times10^{9}\cdot\tok{bits}{b}/8\ \text{bytes}}
Batch-1 decode upper bound: the grey bars in the panel.
Loading figure
Decode throughput of a 7B model on a GPU with 150 TFLOP/s and 1 TB/s (weights only; the KV cache adds bytes per sequence). Drag the cyan dot to batch more sequences per read of the weights: the amber memory-bound line rises with BB until it meets the grey compute ceiling at the knee (B≈150B \approx 150 in FP16, 38 in INT4). At B=1B = 1 FP16 gives 71 tokens/s, INT8 143 and INT4 286: fewer bytes per weight move the whole line up. Past the knee, more batching adds nothing, and tokens per request never rise with BB.

The same reasoning splits a request into two phases. Prefill reads the prompt, hundreds or thousands of tokens that all reuse each weight in one pass, so it is compute bound. Decode then produces one token per sequence per pass, so at small batch it is bandwidth bound, and that is the phase weight quantisation speeds up. Weights are not the whole memory bill either: total inference memory is weights + KV cache + activations. A 7B model in FP16 caches 512 KiB of keys and values per token, 2 GiB per 4096-token sequence. On a 24 GiB GPU, FP16 weights (13 GiB) leave room for five such sequences; INT4 weights (about 3.3 GiB) leave room for ten. Lesson 6.4 sizes and quantises that cache.

W (the viewport's weight matrix)[,]2,048 values

Seeded Gaussian weights; row i has standard deviation between 0.010 and 0.047, a 5x spread of channel scales as in a real projection matrix. Each pillar is one weight.

In the viewport every pillar is a weight x\tok{x}{x}. Every few seconds a wave sweeps across the matrix and each pillar snaps to the nearest allowed value x^\tok{x_hat}{\hat x}; rose slivers show the rounding error, then the originals return. The panel on the right gives, for each bit width, the 7B model's weight memory (amber), the bandwidth bound on decode speed (grey) and the SQNR of this matrix (emerald). At 8 bits with one scale per row the error is invisible: SQNR 45.3 dB, 7.00 GB and 143 tokens/s, against 14 GB and 71 tokens/s in FP16.

Professional Machine Learning Engineer exam guide, Google Cloud, Serving and scaling models

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Ch. 13, computational performance (hardware)

02Uniform affine quantisation

A uniform quantiser covers a real interval with 2b2^{b} evenly spaced values. It stores an integer code qq per weight and two numbers per block of weights: the scale ss (the step between neighbouring values) and the zero-point zz (the code that means 0).

Do it by hand first with five numbers. With 3 bits there are 8 codes, 0 to 7. The numbers run from −0.90 to 0.75, a range of 1.65, and 8 codes have 7 gaps, so each step is 1.65/7=0.2361.65/7 = 0.236. Real 0 sits 0.90/0.236=3.80.90/0.236 = 3.8 steps above the bottom, which rounds to code 4. Then every number goes to its nearest grid point. Drag the cyan dots, change the bits and watch the grid, the arrows and the table follow.

Loading figure
Grey ticks: the 2b2^b representable values x^=s(q−z)\hat x = s(q - z), numbered by code (the amber number is zz). Amber posts: the clipping range [α,β][\alpha, \beta]. Emerald arrows: rounding each number to its nearest tick. Rose bars: the error e=x−x^e = x - \hat x, never more than half a step.
q=clip⁡ ⁣(round⁡(xs)+z,  qmin⁡,  qmax⁡),x^=s (q−z)q=\operatorname{clip}\!\Big(\operatorname{round}\big(\tfrac{\tok{x}{x}}{\tok{scale}{s}}\big)+\tok{zero_point}{z},\;q_{\min},\;q_{\max}\Big),\qquad \tok{x_hat}{\hat x}=\tok{scale}{s}\,(q-\tok{zero_point}{z})
Quantise (round to the nearest code, clip to the code range) and dequantise.
Scale and zero-point from a clipping range1/4
  1. 1
    x∈[α,β] ↦ q∈[qmin⁡,qmax⁡]:q=xs+zx\in[\alpha,\beta]\ \mapsto\ q\in[q_{\min},q_{\max}]:\qquad q=\frac{x}{s}+z

    Map the real clipping range linearly onto the codes. s is the real distance between adjacent codes; z is where real 0 lands.

The viewport is at 3 bits, one scale for the whole tensor, asymmetric. The weights span [α,β]=[−0.117, 0.098]\tok{clip_range}{[\alpha,\beta]}=[-0.117,\ 0.098], so s=0.2155/7=0.0308s=0.2155/7=0.0308 and z=round⁡(0.117/0.0308)=round⁡(3.81)=4z=\operatorname{round}(0.117/0.0308)=\operatorname{round}(3.81)=4: codes 0 to 7, with code 4 meaning 0. In the panel behind the matrix the cyan histogram of x/s+zx/s+z sits on that integer grid between the amber clipping posts, and collapses onto the eight integers as the pillars snap.

Switch the mapping to symmetric at the same 3 bits and one scale.

Weights are roughly symmetric around 0, so at 8 bits the difference is small (40.67 versus 41.40 dB here); asymmetric mapping matters for one-sided data such as activations after ReLU.

Maths, CS & AI Compendium, Henry Ndubuaku, Number formats and quantisation

03Granularity and outliers

One scale has to serve every value in its block, and the block's largest magnitude sets it. Per-tensor uses a single ss. Per-channel uses one ss per output row; it folds out of each dot product, yi=si∑jqijxjy_i=s_i\sum_j q_{ij}x_j, so it costs nothing at inference. Group-wise gives every run of gg consecutive inputs in a row its own scale: 32 × 4 = 128 scales here with g=16g=16, and g=128g=128 is the usual choice for LLM INT4.

s (group-wise scales)[,]128 values

Stored in FP16 next to the codes. Per-channel is the same with one group per row; per-tensor is a single number. g = 128 with FP16 scales adds 16/128 = 0.125 bits per weight.

The outlier switch is on: four isolated weights between 1.18 and 1.65 in magnitude, 10 to 14 times the largest ordinary weight (0.117). With one scale for the tensor, s=1.65/7=0.236\tok{scale}{s}=1.65/7=0.236 at 4 bits, so every ordinary weight lies within half a step of 0: 2044 of the 2048 codes are 0, only the outliers survive and SQNR is 7.4 dB. Switch to per-channel and the damage is confined to the four outlier rows, which stand out in the rose strip on the left (17.0 dB); group-wise confines it to four groups of 16 (24.0 dB).

Replay the snap with a single 4-bit scale for the whole matrix and the outliers on.

Now give every output row its own scale.

Weights are the easy case: they are fixed, so their ranges are known exactly. Activations change with every input and must be quantised on the fly. Large transformers develop outlier features: a few hidden dimensions whose activations are tens of times larger than the rest, consistently across tokens. A per-tensor activation scale is then set by those few channels and flattens everything else, exactly like the outlier switch here. Why not give each activation feature its own scale? The algebra below shows the problem.

Which scales factor out of a matmul1/3
  1. 1
    yi=∑jWij aj,Wij≈si qij ⇒ yi≈si∑jqij ajy_i=\sum_j W_{ij}\,a_j,\qquad W_{ij}\approx s_i\,q_{ij}\ \Rightarrow\ y_i\approx s_i\sum_j q_{ij}\,a_j

    A per-row weight scale s_i does not depend on j, the summed index, so it moves outside the sum: one integer dot product per row, one multiply at the end.

So practical schemes either migrate the range into the weights (SmoothQuant), keep the few outlier dimensions in FP16 and run the rest in INT8 (LLM.int8(), a mixed-precision decomposition), or quantise only the weights.

Maths, CS & AI Compendium, Henry Ndubuaku, Number formats and quantisation

AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services, Deployment and orchestration of ML workflows

The rest of this lesson is in Pro

  • 04Error analysis: s²/12 and 6 dB per bit
  • 05PTQ, QAT, INT4 and FP8
  • 06Other levers and the frontier
  • Python challenge: Write a quantiser
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between INT8 and FP8 quantization?
Both store one byte per value. INT8 is a uniform grid: an integer code times a shared scale, so the absolute step is the same everywhere in the range and outliers force a coarse step. FP8 is a tiny floating-point format with an exponent, so its step grows with the magnitude and small values keep relative precision. INT8 needs careful scale granularity and calibration; FP8 tolerates wider ranges with simple per-tensor scaling and is supported natively by recent data-centre GPUs for training and inference.
Does quantization make a model faster or only smaller?
Both, when the workload is memory-bound. LLM decoding at small batch sizes reads every weight once per token, so halving the bytes per weight roughly doubles the upper bound on tokens per second. Speed-ups also need kernels that work on the quantised format directly; storing INT4 and converting the whole model back to FP16 before running it saves memory but not time. Compute-bound workloads such as large-batch prefill gain mainly when the hardware has faster low-precision arithmetic.
What is the difference between post-training quantization and quantization-aware training?
Post-training quantization rounds a finished model using scales computed from its weights and a small calibration set, with no gradient updates, so it is fast and needs no training pipeline. Quantization-aware training simulates the rounding during fine-tuning and uses the straight-through estimator to pass gradients through it, so the weights learn to tolerate the grid. Use post-training quantization first and switch to quantization-aware training when its accuracy loss is over budget, which is more common at 4 bits and below or on small models.

Next in Production MLOps and Certification Prep

6.2Distribution shift and drift monitoringCovariate, label and concept drift, the statistics that detect them, and when to retrain.

Sources

  • Professional Machine Learning Engineer exam guide, Google Cloud
  • AWS Certified Machine Learning Engineer – Associate exam guide, Amazon Web Services
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola