Skip to content
LearnMLByDoing
Free

3.4 Convolutions, receptive fields and feature maps

34 minProintermediateDLHOMLCompendiumD2L

A convolution slides one small set of weights across an image, so every output cell asks the same local question, and stacking such layers widens the region each cell can see.

01Why convolution: local, shared weights

Start in one dimension. Take one row of a small image, 16 numbers, and a kernel of three weights. Put the kernel over three neighbouring pixels, multiply each pair and add: that is one output number. Slide one pixel to the right and do it again. The widget does this for row 12 of a handwritten 4 with the kernel (−1,0,1)(-1, 0, 1), which computes right pixel minus left pixel.

Loading figure
Press Run or Step. At position 9 the window covers pixels 8, 9 and 10 (values 0, 4/9 and 1), so y[9]=(−1)(0)+(0)(0.44)+(1)(1)=1.00y[9] = (-1)(0) + (0)(0.44) + (1)(1) = 1.00: the row brightens there. Change the weights to (1,1,1)(1, 1, 1) and the kernel sums its neighbourhood instead.

The viewport shows the same digit as a 16 × 16 grid of bars; height and brightness are the pixel intensity of the X\tok{input}{X}, from 0 to 1. An amber 3 × 3 window slides across it. At every stop it multiplies the nine pixels under it by the nine kernel weights W\tok{weights}{W} (listed on the card beside the window and tinted onto its taps), adds the products, and writes one cell of the feature map Y\tok{output}{Y} floating above: emerald bars for positive responses, rose bars for negative ones.

Three assumptions about images are built into that operation. Locality: a useful feature such as an edge depends on a small neighbourhood, so each output reads only a 3 × 3 patch. Weight sharing: an edge detector that works in one corner works everywhere, so every position reuses the same nine weights. Translation equivariance follows from the sharing: shift the digit one pixel to the right and the whole feature map shifts one cell to the right (away from the border).

Parameters, 16 × 16 → 16 × 16[,]

A dense layer connects each of the 256 outputs to each of the 256 pixels. The convolution has 9 weights (plus one bias) whatever the image size. For a 224 × 224 RGB image mapped to 64 channels of the same size, dense needs about 4.8 × 10¹¹ weights; a 3 × 3 convolution needs 1,792 including biases.

The kernel in the viewport is the Sobel x filter. Watch the map fill: an emerald band runs along the left side of each stroke, where intensity rises from left to right, and a rose band along its right side, where it falls. Flat regions, black or white, give 0.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 9.1 and 9.2

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.1, from fully connected layers to convolutions

02One output cell: cross-correlation

In the simplest case, no padding and steps of one pixel, output cell (i,j)(i, j) is the window whose top-left corner sits on pixel (i,j)(i, j):

Y[i,j]=∑u=02∑v=02W[u,v]  X[i+u, j+v]\tok{output}{Y}[i,j]=\sum_{u=0}^{2}\sum_{v=0}^{2}\tok{weights}{W}[u,v]\;\tok{input}{X}[i+u,\ j+v]
3 × 3 kernel, s = d = 1, p = 0. Nine products, one sum.

The general form adds three knobs. Pad the input with p\tok{padding}{p} rings of zeros to get X~\tilde X; move the window s\tok{stride}{s} pixels at a time, so its corner is at (s i, s j)(s\,i,\ s\,j); and space the taps d\tok{dilation}{d} pixels apart:

Y[i,j]=∑u=0k−1∑v=0k−1W[u,v]  X~[s i+d u, s j+d v]+b\tok{output}{Y}[i,j]=\sum_{u=0}^{k-1}\sum_{v=0}^{k-1}\tok{weights}{W}[u,v]\;\tok{input}{\tilde X}\big[\tok{stride}{s}\,i+\tok{dilation}{d}\,u,\ \tok{stride}{s}\,j+\tok{dilation}{d}\,v\big]+b
Discrete 2D cross-correlation with stride s, dilation d and bias b. Indices start at 0; X̃[a, b] = X[a − p, b − p] inside the image and 0 on the padding.
The strongest response, Y[12, 9] = 41/4
  1. 1
    s=d=1, p=1:Y[12,9]=∑u,vW[u,v] X~[12+u, 9+v]=∑u,vW[u,v] X[11+u, 8+v]s=d=1,\ p=1:\quad Y[12,9]=\sum_{u,v}W[u,v]\,\tilde X[12+u,\,9+v]=\sum_{u,v}W[u,v]\,X[11+u,\,8+v]

    Padding shifts the index by one: padded row 12 is image row 11. The widget below does this bookkeeping for any cell.

The HUD's y[i,j]y[i,j] row shows the cell being written and its value. The first row of the map runs slowly so you can follow the window; the right edge of the same stroke gives Y[12,11]=−4Y[12,11]=-4.

Loading figure
The derivation above, for any cell. It opens on Y[12,9]=4Y[12, 9] = 4; click (12,11)(12, 11) for the −4-4, or a cell in row 0 to watch padding zeros enter the sum. Turn on Park the 3D window to stop the scene's window on the same cell.

Deep Learning, Goodfellow, Bengio & Courville, Ch. 9.1, the convolution operation

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.2, convolutions for images

03Output shape: stride, padding and dilation

Three hyperparameters set how many window positions fit. Stride s\tok{stride}{s} is the step between positions. Padding p\tok{padding}{p} adds zeros on every side, so border pixels are visited as often as interior ones. Dilation d\tok{dilation}{d} spreads the kk taps dd pixels apart, so the kernel spans d(k−1)+1d(k-1)+1 pixels with the same k2k^2 weights.

nout=⌊n+2p−d(k−1)−1s⌋+1n_{\text{out}}=\left\lfloor\frac{n+2\tok{padding}{p}-\tok{dilation}{d}(k-1)-1}{\tok{stride}{s}}\right\rfloor+1
Per spatial axis. With s = 1 and d = 1, p = (k − 1)/2 keeps the size ('same' padding).
Counting window positions1/5
  1. 1
    npad=n+2pn_{\text{pad}}=n+2p

    The padded input the window moves over.

The viewport lays every map out in input coordinates: a cell is drawn at the centre of the patch it summarises (the whole layer lifted and set back), so with s=2s=2 the 8 × 8 map has cells two pixels apart, and its last cell is centred on pixel 14, not 15.

Spread the taps: dilation d=2d = 2 with stride 1 and padding 1.

Same dilation, padding raised to 2.

Stride 3 with no padding.

X (a batch of images)[,,,]4,816,896 values

PyTorch's channels-first layout. A 3 × 3 convolution with stride 2 and padding 1 maps it to 32 × 64 × 112 × 112.

W (Conv2d weight)[,,,]1,728 values

One k × k × C_in filter per output channel, 1,728 weights plus 64 biases. Each output channel sums over all input channels; the lab's single-channel case is C_in = C_out = 1.

Loading figure
What a channel sum looks like with numbers: one 3 × 3 × 3 filter on an RGB patch gives one output value. Then count a whole layer's parameters: k2CinCout+Coutk^2 C_{\text{in}} C_{\text{out}} + C_{\text{out}}.

Dive into Deep Learning, Zhang, Lipton, Li & Smola, Sec. 7.3, padding and stride

Hands-On Machine Learning, Aurélien Géron, Ch. 14, convolutional layers

The rest of this lesson is in Pro

  • 04Classic kernels and learned filters
  • 05Stacking: the receptive field grows
  • 06Modern practice: 1 × 1, depthwise, residual, ViT
  • Python challenge: Write conv2d, its output size and its receptive field
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

How do you calculate the output size of a convolutional layer?
For input size n, kernel size k, stride s, padding p on each side and dilation d, the output size per axis is floor((n + 2p - d(k - 1) - 1) / s) + 1. For example, a 224 pixel input with a 3 by 3 kernel, stride 2 and padding 1 gives floor(223 / 2) + 1 = 112. With stride 1 and no dilation, padding (k - 1) / 2 keeps the size unchanged. The same formula applies to pooling layers.
What is the receptive field of a CNN?
The receptive field of a unit is the region of the input image that can influence its value. A single 3 by 3 layer sees 3 by 3 pixels; each further layer adds (k - 1) times its dilation times the product of all earlier strides, so four stacked 3 by 3 layers with stride 1 see 9 by 9 pixels. Strides, pooling and dilation make it grow faster. In practice the pixels near the centre of the field have much more influence than those at its edge.
Is convolution in deep learning actually cross-correlation?
Yes. Mathematical convolution flips the kernel before sliding it, while the Conv2d layers in PyTorch, TensorFlow and other libraries slide it unflipped, which is cross-correlation. Because the kernel weights are learned, the difference does not matter for training: the network would learn the flipped weights. It matters when you use a hand-designed asymmetric kernel such as an edge detector, where a flip reverses the sign of the response.

Next in Deep Learning and Autograd

3.5Recurrent networks, LSTMs and backprop through timeUnroll a recurrent cell through time, watch gradients vanish down the chain, and see how gating keeps a memory alive.

Sources

  • Deep Learning, Goodfellow, Bengio & Courville
  • Hands-On Machine Learning, Aurélien Géron
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Dive into Deep Learning, Zhang, Lipton, Li & Smola