How convolution works in a CNN
What a convolution actually computes, shown cell by cell: kernels, feature maps, stride, padding, channels, pooling, and how stacked layers see larger and larger patterns.
A convolutional neural network (CNN) is built around one small operation, repeated many thousands of times: lay a small grid of weights over a patch of an image, multiply the overlapping numbers, and add them up. Everything else, from stride and padding to pooling and receptive fields, is bookkeeping around that step.
This guide works through the operation with real numbers you can check by hand. Every output value, shape and parameter count below has been computed, and the PyTorch script near the end reproduces them.
An image is a grid of numbers
To a computer, a greyscale image is a rectangle of numbers, one per pixel. In the common 8-bit format each number runs from 0 to 255. Figure 1 shows a tiny 8 × 8 image of a handwritten 7, stored the way the MNIST digit dataset stores its images: 0 is background and 255 is full-strength ink.
A colour image is three of these grids stacked together, one each for red, green and blue. Each grid is called a channel, so a 224 × 224 colour photo is a block of 224 × 224 × 3 = 150,528 numbers. Before training, those numbers are usually scaled to the range 0 to 1 or standardised to a mean of 0. The examples in this article use small whole numbers instead, so you can follow the arithmetic.
Why not just use a dense layer?
The simplest neural network layer is a dense (fully connected) layer: every input connects to every output through its own weight. To feed an image into one, you flatten the grid into a long list. That works for tiny images, but it scales badly and ignores what makes an image an image.
- Too many parameters. A dense layer from a 224 × 224 × 3 image to 1,000 units needs 150,528 × 1,000 = 150,528,000 weights plus 1,000 biases: 150,529,000 parameters in the first layer alone.
- No sense of neighbourhood. Once the image is flattened, the layer has no idea which pixels sit next to each other. Shuffle every image’s pixels in the same fixed order and a dense layer could learn from the scrambled images just as well, which shows it isn’t using the layout at all. Yet neighbouring pixels are where the information is: edges, corners and textures are all local patterns.
- No sense of position. A cat in the top-left corner and the same cat in the bottom-right hit completely different weights. The layer has to learn “cat” separately at every location, from examples of cats at every location.
Convolution fixes all three by building two assumptions into the layer. First, each output looks only at a small neighbourhood of the input. Second, a pattern worth detecting in one place is worth detecting everywhere, so the same weights are reused at every position. The rest of this article shows what those two ideas look like in practice.
One convolution step
A convolutional layer keeps its weights in a kernel, also called a filter: a small grid of numbers, most often 3 × 3. To compute one output value, place the kernel over a patch of the image the same size as the kernel, multiply each pixel by the weight on top of it, add up the nine products, and add one extra learned number called the bias.
Written out row by row, the step in Figure 2 is:
That single number is a match score. The kernel has positive weights in its top-left corner and negative weights in its bottom-right, and its weights add up to 0, so it gives 0 (plus the bias) on a flat patch, a large positive value when the top-left of the patch is brighter than the bottom-right, and a negative value for the opposite. It is a detector for one kind of diagonal edge. In this patch the pixels under the negative weights (2, 2 and 1) are brighter than those under the positive weights (0, 1 and 1), so the score is negative.
Sliding the kernel: the feature map
One step gives one number. To process the whole image, slide the kernel across it one pixel at a time, left to right and top to bottom, doing the same multiply-and-add at every position. On a 6 × 6 image a 3 × 3 kernel fits in 4 positions across and 4 down, so the output is a 4 × 4 grid of 16 values.
Figure 3 highlights the window in the top row, third column. The same arithmetic gives:
That 9 is the largest value in the output: in this patch the bright pixels sit under the kernel’s positive weights and the dim ones under its negative weights. The output grid is called a feature map (or activation map). It is a map of where in the image the kernel’s pattern appears, and how strongly.
Step through all 16 positions to watch the feature map fill in:
0·2 + 1·1 + 3·0 + 1·1 + 2·0 + 2·(−1) + 0·0 + 2·(−1) + 1·(−2) = −4, plus bias 1 = −3
Two facts about this computation drive everything that follows. Each output depends on only nine input pixels, its local neighbourhood. And all 16 outputs are computed with the same ten numbers: nine weights and one bias. Sliding the kernel is how one small set of weights gets reused across the whole image.
What a kernel detects
The weights in a kernel decide which pattern it responds to. The easiest pattern to see is an edge, a place where brightness changes sharply. Figure 4 uses an image whose left half is bright (9) and whose right half is dark (1), with a vertical edge down the middle.
The vertical-edge kernel has a column of 1s on the left and a column of −1s on the right, so each output is “left column minus right column”, summed over three rows. Over the flat bright region every window gives 9 − 9 = 0 per row, and over the flat dark region 1 − 1 = 0. The two windows that straddle the edge give 9 − 1 = 8 per row, and 3 × 8 = 24. The feature map is a two-column-wide line of 24s exactly where the edge is.
The horizontal-edge kernel is the same kernel turned on its side. It compares the top row of each window with the bottom row, and in this image those are always equal, so its feature map is zero everywhere. A kernel responds to its own pattern and ignores others. The sign carries information too: mirror the image so the dark side is on the left, and the vertical-edge kernel gives −24 instead of 24.
Image-processing software has used hand-designed kernels like these for decades:
| Kernel | Weights, row by row | What it does |
|---|---|---|
| Vertical edge | 1 0 −1 / 1 0 −1 / 1 0 −1 | Responds where brightness changes from left to right |
| Sobel (vertical) | 1 0 −1 / 2 0 −2 / 1 0 −1 | The same, with extra weight on the middle row, which smooths out some noise |
| Horizontal edge | 1 1 1 / 0 0 0 / −1 −1 −1 | Responds where brightness changes from top to bottom |
| Box blur | every weight 1/9 | Replaces each pixel with the mean of its 3 × 3 neighbourhood |
| Sharpen | 0 −1 0 / −1 5 −1 / 0 −1 0 | Exaggerates the difference between a pixel and its neighbours |
A handy check: the edge kernels’ weights add up to 0, so they give 0 on any flat patch. The blur and sharpen kernels’ weights add up to 1, so a flat patch passes through unchanged.
The point of a CNN is that nobody writes these kernels by hand. The weights start as small random numbers and are adjusted by gradient descent, using gradients from backpropagation, towards whatever values reduce the network’s error. When the first-layer kernels of trained image networks are visualised, many of them do end up looking like oriented edge detectors and colour-blob detectors, which suggests edges are a useful place to start.
Stride, padding and output size
Sliding a 3 × 3 kernel over a 6 × 6 image gave a 4 × 4 output. The image shrank by two pixels in each direction because the kernel can’t hang off the edge. That causes two problems. Stack a few layers and the image shrinks every time. And border pixels are used less than central ones: on the 6 × 6 image a corner pixel falls inside only 1 of the 16 windows, while a pixel two steps in from the corner falls inside 9.
Padding
Padding adds a border of extra pixels, almost always zeros, around the input before convolving. With a 3 × 3 kernel, a border one pixel wide (p = 1) makes the output the same size as the input. This setting is called “same” padding. No padding at all is called “valid” padding, because every window lies entirely on real pixels.
In the left panel of Figure 5 the first window now sits over the corner, mostly on padding. Only four real pixels (0, 1, 1 and 2) fall under it, giving 0·0 + 1·(−1) + 1·(−1) + 2·(−2) + 1 = −5. The inner 4 × 4 of the padded output is exactly the unpadded feature map from Figure 3; padding adds a ring of border values around it.
For an odd kernel size k and stride 1, “same” padding is p = (k − 1)/2: 1 for a 3 × 3 kernel, 2 for a 5 × 5. This is one reason odd kernel sizes are the norm: they have a centre pixel and pad evenly on both sides.
Stride
The stride is how far the kernel moves between positions. Everything so far used stride 1. With stride 2 the kernel jumps two pixels at a time, so the output is roughly half the size in each direction. In the right panel of Figure 5, the 6 × 6 image with stride 2 gives a 2 × 2 output of −3, 9, 3 and 4: the stride-1 outputs from every second row and column. The windows cover rows 1 to 3 and 3 to 5, so the sixth row and column are never used. When the stride doesn’t divide evenly, the remainder is simply dropped. Strided convolutions are a common way to shrink a feature map on purpose, as an alternative to the pooling described below.
The output-size formula
The top line, n + 2p − k, is how far the kernel can travel across the padded input. Dividing by the stride counts the jumps, and the + 1 counts the starting position. Some worked examples:
| n | k | p | s | Calculation | Output |
|---|---|---|---|---|---|
| 6 | 3 | 0 | 1 | (6 − 3)/1 + 1 | 4 (Figure 3, “valid”) |
| 6 | 3 | 1 | 1 | (6 + 2 − 3)/1 + 1 | 6 (“same”) |
| 6 | 3 | 0 | 2 | ⌊3/2⌋ + 1 = 1 + 1 | 2 |
| 6 | 3 | 1 | 2 | ⌊5/2⌋ + 1 = 2 + 1 | 3 |
| 224 | 3 | 1 | 1 | (224 + 2 − 3)/1 + 1 | 224 |
| 224 | 7 | 3 | 2 | ⌊223/2⌋ + 1 = 111 + 1 | 112 |
The last row is the first layer of the ResNet family of image classifiers, which turns a 224 × 224 photo into a 112 × 112 feature map. Height and width use the same formula separately, so non-square images and kernels work the same way. In PyTorch, padding='same' is accepted only when the stride is 1; for strided layers you give the padding as a number.
Channels, filters and weight sharing
Real inputs have more than one channel. A colour image has three, and the feature maps inside a network often have 64, 256 or more. A convolution handles this by making the kernel as deep as its input. On an RGB image, a 3 × 3 filter is really a 3 × 3 × 3 block of 27 weights, one 3 × 3 slice per channel.
At each position, every slice is multiplied with its own channel’s patch, and all 27 products are added together with a single bias. The result is still one number, so one filter produces one output channel, however many input channels it reads:
A layer uses many filters, each with its own weights, and stacks their feature maps. The number of filters becomes the number of output channels. Figure 6 shows a layer with four filters on a 32 × 32 RGB image, with padding 1 so the height and width stay at 32.
Counting parameters
Each filter has k × k × Cin weights and one bias, and there are Cout filters:
The layer in Figure 6 has (3 · 3 · 3 + 1) · 4 = 112 parameters. A typical first layer for 224 × 224 photos uses 64 filters: (3 · 3 · 3 + 1) · 64 = 1,792 parameters, and with padding 1 its output is 224 × 224 × 64. A second 3 × 3 layer that reads those 64 channels and produces 64 more has (3 · 3 · 64 + 1) · 64 = 36,928.
Notice what the formula leaves out: the image size. The same 1,792 weights process a 32 × 32 thumbnail or a 4,000 × 3,000 photo; only the size of the output changes.
Weight sharing
That independence comes from weight sharing: every position in the image is processed with the same filters. To see how much it saves, compare the 1,792-parameter layer with a dense layer that maps the same input (224 × 224 × 3 = 150,528 values) to the same output (224 × 224 × 64 = 3,211,264 values). The dense layer needs a weight for every input–output pair: 150,528 × 3,211,264 = 483,385,147,392 weights, roughly 483 billion.
Weight sharing saves parameters, not arithmetic. Each of those 3,211,264 outputs still needs 27 multiply-adds, 86,704,128 in total, so convolutional layers are cheap to store but still expensive to run.
Translation equivariance
Weight sharing has a second benefit. Because the same filter is applied everywhere, shifting the input shifts the output by the same amount: if a pattern moves one pixel right, its response moves one cell right and keeps its value. This property is called translation equivariance.
This is exactly what the dense layer lacked. A filter that has learned to detect a diagonal stroke detects it anywhere in the image, without needing training examples of the stroke at every position. Two caveats: the property is exact only away from the borders, where padding changes what the kernel sees, and strided layers and pooling make it approximate, because a one-pixel shift no longer lines up with a coarser output grid.
Equivariance is not the same as invariance. An equivariant feature map moves when the input moves; an invariant output doesn’t change at all. For classifying a whole image you want the final answer, “this is a cat”, to be invariant to where the cat is. CNNs get there gradually, through pooling and through deeper layers that summarise larger areas.
ReLU and pooling
A convolution is linear: it only multiplies and adds. Stacking linear operations gives another linear operation; two 3 × 3 convolutions in a row are equivalent to a single 5 × 5 convolution, so without something in between, extra layers add nothing. Every convolutional layer is therefore followed by a nonlinear activation function, most often the rectified linear unit:
ReLU keeps positive values and replaces negative ones with zero. Applied to the feature map from Figure 3, the five negative values (−3, −1, −5, −2 and −5) become 0 and everything else passes through. In detector terms, a negative score means “the pattern isn’t here”, and ReLU turns that into a plain no.
Max pooling
A pooling layer shrinks a feature map by summarising small blocks of it. The most common version, 2 × 2 max pooling with stride 2, splits the map into non-overlapping 2 × 2 blocks and keeps the largest value in each.
For the running example, the four blocks give:
Pooling does three useful things. It cuts the number of values to a quarter, which reduces the work in later layers. It keeps the strongest response in each block, answering “did the pattern appear around here?” rather than “exactly where?”. And because of that, nudging a pattern by a pixel within a block often leaves the pooled value unchanged, which adds a little of the invariance described above. Pooling has no parameters to learn.
There are alternatives. Average pooling keeps the mean instead of the maximum, and many modern networks skip pooling layers and shrink feature maps with stride-2 convolutions instead. Global average pooling, which averages each whole channel down to a single number, is a common way to finish a network just before the final classifier.
Stacking layers into a CNN
Receptive fields
One 3 × 3 layer sees a 3 × 3 patch of its input. The interesting part happens when layers are stacked. A unit in the second layer looks at a 3 × 3 patch of the first layer’s feature map, and each of those nine values looked at its own 3 × 3 patch of the image. Together they cover 5 × 5 pixels, and a third layer reaches 7 × 7. The patch of the original image that a unit depends on is called its receptive field.
With stride-1 layers of kernel size k, the receptive field after L layers is 1 + L(k − 1): 3, 5, 7, 9 and so on for 3 × 3 kernels. Stride and pooling make it grow much faster, because each step on a coarser grid spans several input pixels. Applied layer by layer, the general rule is:
For the small network in Figure 10 below, the receptive field is 3 after the first convolution, 4 after the first pooling, 8 after the second convolution (4 + 2 · 2) and 10 after the second pooling. Each value in the final 8 × 8 feature map summarises a 10 × 10 patch of the 32 × 32 input.
This is also why stacks of 3 × 3 layers became the standard design, popularised by the VGG networks. Two 3 × 3 layers see a 5 × 5 region using 2 × 9 = 18 weights per pair of input and output channels, where a single 5 × 5 layer would use 25. Three 3 × 3 layers see 7 × 7 with 27 weights instead of 49. The stack also gets a ReLU after every layer, so it can represent more complicated functions than one big linear filter.
From edges to objects
Because receptive fields grow with depth, different layers end up detecting patterns at different scales. Visualisations of trained image classifiers show a broad progression:
- Early layers respond to edges, colour contrasts and simple gradients, the kind of pattern in Figure 4.
- The next layers combine edges into textures and simple shapes: stripes, grids, corners and curves.
- Middle layers respond to parts: an eye, a wheel, a patch of fur, a printed letter.
- Late layers respond to whole objects or large parts of them, such as faces or the fronts of cars, across a range of positions and sizes.
Nobody programs this hierarchy. It emerges because each layer can only build on what the layer below found, over a slightly larger area, and because doing so reduces the training error. The boundaries are fuzzy, and the exact features depend on the data and the architecture.
A small CNN, end to end
Figure 10 puts the pieces together into a small classifier for 32 × 32 colour images in 10 classes, the format of the CIFAR-10 benchmark dataset. Two blocks of convolution, ReLU and pooling turn the image into a stack of feature maps; a flatten step and one dense layer turn those into class scores.
Follow the shapes down the figure. Padding 1 keeps each convolution’s height and width unchanged, and each pooling layer halves them: 32 → 16 → 8. Meanwhile the channel count grows from 3 to 16 to 32, so the network trades spatial detail for a richer description of each location. Flatten unrolls the final 8 × 8 × 32 block into 2,048 numbers, and the dense layer maps those to 10 scores, one per class. Softmax turns the scores into probabilities that add up to 1.
The parameter counts tell a story. The two convolutional layers have 448 and 4,640 parameters, while the single dense layer has 20,490, about 80% of the 25,578 total. Convolutions are cheap in parameters; dense layers on flattened feature maps are not. That is one reason many modern architectures replace the flatten-and-dense step with global average pooling.
Checking the numbers in PyTorch
PyTorch’s nn.Conv2d and F.conv2d implement everything above. PyTorch orders image tensors as (batch, channels, height, width), so one 32 × 32 RGB image has shape (1, 3, 32, 32), with the channels first rather than last as in this article’s figures. This script reproduces the running example and builds the network from Figure 10:
import torch
import torch.nn as nn
import torch.nn.functional as F
# The running example: one 6×6 image, one 3×3 kernel, bias 1.
# PyTorch wants (batch, channels, height, width).
x = torch.tensor([[0, 1, 3, 2, 0, 3],
[1, 2, 2, 2, 1, 1],
[0, 2, 1, 1, 0, 2],
[3, 1, 1, 1, 0, 3],
[3, 2, 0, 1, 0, 3],
[3, 1, 1, 3, 0, 3]], dtype=torch.float32).view(1, 1, 6, 6)
k = torch.tensor([[2, 1, 0],
[1, 0, -1],
[0, -1, -2]], dtype=torch.float32).view(1, 1, 3, 3)
b = torch.tensor([1.0])
z = F.conv2d(x, k, b) # no flip: cross-correlation
print(z[0, 0].int().tolist())
print(F.max_pool2d(F.relu(z), 2)[0, 0].int().tolist())
print(F.conv2d(x, k, b, padding=1).shape) # "same"
print(F.conv2d(x, k, b, stride=2).shape)
# The small CNN from Figure 10, on one 32×32 RGB image.
model = nn.Sequential(
nn.Conv2d(3, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(),
nn.Linear(32 * 8 * 8, 10),
)
h = torch.randn(1, 3, 32, 32)
for layer in model:
h = layer(h)
print(f"{type(layer).__name__:<10}{tuple(h.shape)}")
print(sum(p.numel() for p in model.parameters()))
print(nn.Conv2d(3, 64, kernel_size=3).weight.shape)It prints:
[[-3, 3, 9, 2], [1, 5, 7, -1], [3, 4, 4, -5], [8, -2, 1, -5]]
[[5, 9], [8, 4]]
torch.Size([1, 1, 6, 6])
torch.Size([1, 1, 2, 2])
Conv2d (1, 16, 32, 32)
ReLU (1, 16, 32, 32)
MaxPool2d (1, 16, 16, 16)
Conv2d (1, 32, 16, 16)
ReLU (1, 32, 16, 16)
MaxPool2d (1, 32, 8, 8)
Flatten (1, 2048)
Linear (1, 10)
25578
torch.Size([64, 3, 3, 3])The first two lines are the feature map from Figure 3 and the pooled result from Figure 8. The two torch.Size lines confirm the output-size formula for “same” padding (6 × 6) and for stride 2 (2 × 2). The per-layer shapes match Figure 10, and 25,578 matches its parameter total. The last line is the weight tensor of a 64-filter first layer: 64 filters of 3 × 3 × 3, which with 64 biases gives the 1,792 parameters from the section on channels.
The model stops at the dense layer, without a softmax, because PyTorch’s nn.CrossEntropyLoss expects raw scores and applies the softmax itself during training. The random input only matters for its shape; the shapes and the parameter count don’t depend on the values.
Summary
- A convolution slides a small kernel over the input. At each position it multiplies the overlapping values, adds them up and adds a bias, giving one value of a feature map.
- What deep learning calls convolution is cross-correlation, with no kernel flip. Because kernels are learned, the difference doesn’t matter.
- The kernel’s weights decide what it detects: edges, blur and sharpening can be written by hand, but a CNN learns its kernels by gradient descent.
- Output size is ⌊(n + 2p − k)/s⌋ + 1. With stride 1, padding (k − 1)/2 keeps the size (“same”); stride 2 roughly halves it.
- A filter spans every input channel and produces one output channel. A layer has (k · k · Cin + 1) · Cout parameters, whatever the image size.
- Weight sharing gives convolutions few parameters and translation equivariance: move the input and the feature map moves with it.
- ReLU after each convolution supplies the nonlinearity; max pooling shrinks feature maps and adds a little tolerance to small shifts.
- Stacking layers grows the receptive field (3 → 5 → 7 for 3 × 3 layers, faster with pooling or stride), so deeper layers detect larger, more abstract patterns.
Further reading on this site
- How backpropagation works, step by step: how the kernel weights in this article are learned from data.
- How support vector machines work: another classic classifier, and a different meaning of the word “kernel”, a similarity function rather than a grid of weights.