How backpropagation works, step by step
Backpropagation explained with a computational graph and a fully worked numeric example: the forward pass, the chain rule, gradients flowing backwards, and one step of gradient descent.
Training a neural network means adjusting its weights until its predictions are good. To know which way to move each weight, and by how much, you need one number per weight: the derivative of the loss with respect to that weight. Backpropagation is the algorithm that produces all of those derivatives in one sweep, for a small constant multiple of the cost of running the network forward.
This explainer builds it up from nothing: a single neuron drawn as a computational graph, the chain rule, a two-layer network worked through with real numbers, one step of gradient descent, then the matrix form that libraries use and the problems that appear in deep networks. All you need is the idea of a derivative as a rate of change.
First, the problem backpropagation solves. The most direct way to estimate a derivative is to measure it: nudge one weight by a tiny amount h, run the network again, and see how much the loss moved.
This works, and it is a handy way to test a gradient calculation, but every weight needs its own extra forward pass. A network with P weights needs P + 1 forward passes to get one full gradient, and real networks have anywhere from thousands to billions of weights. It is also only approximate: h must be small enough to be accurate but large enough that floating-point rounding does not swamp the difference.
Backpropagation gets the exact gradient for every weight from one forward pass and one backward pass. The rest of this article shows how.
Computational graphs and the forward pass
Any calculation can be drawn as a graph in which values flow along edges and each node applies one simple operation. Take a single artificial neuron with one input. It multiplies the input by a weight, adds a bias, and squashes the result with the sigmoid function:
To train it we need a loss: a single number that says how wrong the output is. We will use the squared error with a factor of a half, which makes its derivative tidy:
Now give everything a value: input x = 2.0, weight w = 0.4, bias b = 0.2, and target y = 1.0. Breaking the neuron into one node per operation, the forward pass computes, from left to right:
- p = w · x = 0.4 × 2.0 = 0.8
- z = p + b = 0.8 + 0.2 = 1.0
- a = σ(1.0) = 0.7311
- L = ½(0.7311 − 1)² = 0.0362
Splitting w · x + b into separate multiply and add nodes may look fussy, but it is the whole trick. Every node is now so simple that its derivative is obvious, and backpropagation works entirely by combining these simple local derivatives. Notice too that the forward pass keeps every intermediate value (p, z and a). The backward pass will need them.
The chain rule, intuitively
A derivative is a rate of exchange. Saying ∂u/∂x = 3 means that, near the current point, a small change in x produces a change in u three times as big. The chain rule says what happens when rates are chained together, and it comes in two parts.
Along a path, rates multiply. If u moves three times as much as x, and f moves twice as much as u, then f moves six times as much as x. Nudge x by 0.01 and u moves by 0.03, which moves f by 0.06.
When paths split, their contributions add. If x reaches f by two routes, a nudge to x travels down both and the effects at f sum. With u = 3x, v = −x and f = 2u + 4v:
That is all the calculus backpropagation needs. The difficulty is bookkeeping: in a real network there are astronomically many paths from an early weight to the loss, and summing over them one by one would be hopeless.
Backpropagation avoids this by working backwards from the loss and storing, at every value, a single number: the derivative of the loss with respect to that value. That number already sums over every path from the value to the loss, so anything upstream only needs to multiply by one more local rate. Shared path segments are computed once and reused. In computer-science terms, it is dynamic programming applied to the chain rule.
Backward through a single neuron
Back to the neuron. The backward pass visits the nodes in reverse order. It starts at the loss with ∂L/∂L = 1 (nudging L changes L by the same amount) and applies one rule at every node:
Working from right to left:
- Loss node. For L = ½(a − y)², the local derivative is ∂L/∂a = a − y = 0.7311 − 1 = −0.2689.
- Sigmoid node. The sigmoid has the convenient derivative σ′(z) = σ(z)(1 − σ(z)) = a(1 − a) = 0.7311 × 0.2689 = 0.1966, computed from the stored output. So ∂L/∂z = −0.2689 × 0.1966 = −0.0529.
- Add node. Both local derivatives are 1, so an add node passes the gradient unchanged to each input: ∂L/∂p = ∂L/∂b = −0.0529.
- Multiply node. For p = w · x, the derivative with respect to each input is the other input. So ∂L/∂w = −0.0529 × 2.0 = −0.1058 and ∂L/∂x = −0.0529 × 0.4 = −0.0212.
Read the result. ∂L/∂w = −0.1058 is negative, so increasing w would decrease the loss. That makes sense: the output 0.7311 is below the target of 1, and a larger weight pushes z and therefore a upwards. The size of the number says how sensitive the loss is to this weight right now.
You can check it with a finite difference. Nudging w from 0.4 to 0.4001 changes the loss from 0.03616474 to 0.03615417, a slope of about −0.1057. That agrees with the exact −0.1058 up to the finite difference’s own approximation error, and notice how many decimal places were needed to see the change at all.
The backward pass also produced ∂L/∂x = −0.0212. For a lone neuron that is useless, since we cannot change the input. In a network, though, one layer’s input is the previous layer’s output, and this is exactly the number that gets passed further back.
Step 1 of 10 · forward pass. Start with the given numbers: input x = 2.0, weight w = 0.4, bias b = 0.2 and target y = 1.0.
A worked example with a small network
Now a real network: two inputs, a hidden layer of two sigmoid units and a single sigmoid output. It has six weights and three biases, nine parameters in all. The weights w1 to w4 connect the inputs to the hidden units, w5 and w6 connect the hidden units to the output, and each unit has its own bias. The input is x = (1.0, 0.5) and the target is y = 1.0, with the same loss as before.
The forward pass
Each unit computes a weighted sum z of its inputs plus its bias, then applies the sigmoid. Every value is stored for the backward pass.
| Quantity | Calculation | Value |
|---|---|---|
| z₁ | w₁x₁ + w₂x₂ + b₁ = 0.5 × 1.0 + (−0.4) × 0.5 + 0.1 | 0.4000 |
| h₁ | σ(z₁) | 0.5987 |
| z₂ | w₃x₁ + w₄x₂ + b₂ = 0.3 × 1.0 + 0.8 × 0.5 + (−0.1) | 0.6000 |
| h₂ | σ(z₂) | 0.6457 |
| z₃ | w₅h₁ + w₆h₂ + b₃ = 0.9 × 0.5987 + (−0.5) × 0.6457 + 0.2 | 0.4160 |
| o | σ(z₃) | 0.6025 |
| L | ½(o − y)² = ½(0.6025 − 1)² | 0.0790 |
Every number in this article was computed at full precision and rounded to four decimal places for display, so if you redo a step from the rounded figures the last digit can occasionally differ by one.
The backward pass: one δ per unit
For networks it pays to track one quantity per unit: δ (delta), the gradient of the loss with respect to the unit’s weighted sum z. Once a unit’s δ is known, every gradient around it is one multiplication away.
Output unit. This is the single-neuron calculation again: the loss derivative times the sigmoid derivative.
Hidden units. h1 affects the loss only through z3, and ∂z3/∂h1 = w5. So the gradient arriving at h1 is w5δ3, and multiplying by the sigmoid derivative turns it into δ1. The same goes for h2 with w6.
Here 0.2403 and 0.2288 are h1(1 − h1) and h2(1 − h2), read straight from the stored forward values. Notice the sign of δ2: it is positive because w6 is negative. Raising h2 would lower the output, which is already too low, so the loss would go up.
If a hidden unit fed several units in the next layer, its incoming gradient would be the sum of wkδk over all of them. That is the “add over paths” half of the chain rule doing its job.
Every gradient
With the δs in hand, each weight’s gradient is the δ of the unit it feeds into, times the value travelling along it. Each bias gradient is simply its unit’s δ, because the bias enters z with a local derivative of 1.
| Parameter | Rule | Calculation | Gradient |
|---|---|---|---|
| w₁ | δ₁ · x₁ | −0.0206 × 1.0 | −0.0206 |
| w₂ | δ₁ · x₂ | −0.0206 × 0.5 | −0.0103 |
| w₃ | δ₂ · x₁ | 0.0109 × 1.0 | 0.0109 |
| w₄ | δ₂ · x₂ | 0.0109 × 0.5 | 0.0054 |
| w₅ | δ₃ · h₁ | −0.0952 × 0.5987 | −0.0570 |
| w₆ | δ₃ · h₂ | −0.0952 × 0.6457 | −0.0615 |
| b₁ | δ₁ | −0.0206 | |
| b₂ | δ₂ | 0.0109 | |
| b₃ | δ₃ | −0.0952 |
Nine gradients from one forward pass and one backward pass. If you nudge any one of these parameters by a tiny amount and rerun the forward pass, the change in the loss matches these values. Comparing against finite differences like this, on a small example, is called gradient checking, and it is the standard way to test a hand-written backward pass.
One step of gradient descent
The gradient points in the direction that increases the loss fastest, so gradient descent takes a small step the other way. The step size is set by the learning rate η (eta).
With η = 0.5, every parameter moves by −0.5 times its gradient:
| Parameter | Old value | Gradient | Change (−η × gradient) | New value |
|---|---|---|---|---|
| w₁ | 0.5000 | −0.0206 | +0.0103 | 0.5103 |
| w₂ | −0.4000 | −0.0103 | +0.0051 | −0.3949 |
| w₃ | 0.3000 | 0.0109 | −0.0054 | 0.2946 |
| w₄ | 0.8000 | 0.0054 | −0.0027 | 0.7973 |
| w₅ | 0.9000 | −0.0570 | +0.0285 | 0.9285 |
| w₆ | −0.5000 | −0.0615 | +0.0307 | −0.4693 |
| b₁ | 0.1000 | −0.0206 | +0.0103 | 0.1103 |
| b₂ | −0.1000 | 0.0109 | −0.0054 | −0.1054 |
| b₃ | 0.2000 | −0.0952 | +0.0476 | 0.2476 |
Running the forward pass again with the new parameters, the output rises from 0.6025 to 0.6241 and the loss falls from 0.0790 to 0.0707. The step worked.
The gradient even predicts roughly how much the loss should fall. To first order, a step of −η times the gradient lowers the loss by η times the sum of the squared gradients: 0.5 × 0.0173 = 0.0087. The actual fall is 0.0083, slightly less, because the loss surface curves away from its tangent over the length of the step. The smaller the step, the closer the prediction gets, which is one way to see why the learning rate cannot be made arbitrarily large.
Training is this loop repeated: forward pass, backward pass, update. After 100 steps the output has climbed to 0.9213 and the loss is down to 0.0031. Progress slows as the output nears 1, partly because the loss gradient (o − y) shrinks and partly because the sigmoid flattens out, a first glimpse of a problem we will meet again.
The learning rate is a trade-off. Too small and training crawls; too large and a step can overshoot the valley it was aiming for, so the loss bounces around or blows up. In real training each step also uses a different batch of examples, which we will come to shortly.
The general recipe and the matrix form
Nothing in the worked example depended on the network being small. Write layer l’s weights as a matrix W(l) whose row j holds the weights into unit j, its biases as a vector b(l), and its outputs as a(l), with a(0) = x. For a network with N layers, the forward pass is
and backpropagation is four steps:
- Forward. Compute and store z(l) and a(l) for every layer, then the loss.
- Output layer. δ(N) = ∂L/∂a(N) ⊙ σ′(z(N)), where ⊙ means element-by-element multiplication.
- Hidden layers, from l = N − 1 down to 1: δ(l) = (W(l+1))T δ(l+1) ⊙ σ′(z(l)).
- Gradients. ∂L/∂W(l) = δ(l) (a(l−1))T and ∂L/∂b(l) = δ(l).
The transpose is the “add over paths” rule for a whole layer at once. Entry j of (W(l+1))Tδ(l+1) sums, over every unit k in the next layer, the weight from j to k times δk. The outer product δaT builds a matrix the same shape as W in which entry (j, i) is δj times input i, which is exactly the “δto × afrom” rule.
Plug in the example. W(1) = [[0.5, −0.4], [0.3, 0.8]] and W(2) = [0.9, −0.5]. Then δ(2) = −0.0952, (W(2))Tδ(2) = [−0.0857, 0.0476], and multiplying element-wise by σ′(z(1)) = [0.2403, 0.2288] gives δ(1) = [−0.0206, 0.0109]. The outer product with a(0) = [1.0, 0.5] is
A useful habit when implementing this: check shapes. δ(l) has one entry per unit in layer l, a(l−1) has one per unit in the layer before, so their outer product must have exactly the shape of W(l). A gradient whose shape does not match its parameter is always a bug.
Why it costs about the same as a forward pass
Count the work for one dense layer. The forward pass does one matrix–vector product, W(l)a(l−1). The backward pass does two products of the same size: (W(l))Tδ(l) to pass the gradient back, and δ(l)(a(l−1))T to get the weight gradient. So the backward pass costs roughly twice the forward pass, and a full gradient costs about three forward passes, whether the network has nine weights or nine billion. Finite differences would need one forward pass per weight.
The price is memory. Every a(l) and z(l) from the forward pass must be kept until the backward pass reaches that layer, which is why training needs far more memory than running a trained model. Techniques such as gradient checkpointing trade some of that memory for extra computation by recomputing activations during the backward pass instead of storing them all.
Reverse-mode automatic differentiation
Backpropagation is a special case of a general technique called reverse-mode automatic differentiation. There is also a forward mode, and the difference explains why neural networks use the reverse one.
Forward mode
- Carries derivatives forward alongside the values.
- One pass gives the derivative of every output with respect to one input.
- Efficient when there are few inputs and many outputs.
Reverse mode (backpropagation)
- Runs forward to store values, then sweeps back from the output.
- One pass gives the derivative of one output with respect to every input.
- Efficient when there are many inputs and one output.
Training has millions of inputs to differentiate with respect to (the parameters) and exactly one output (the scalar loss). That is the ideal case for reverse mode, and it is why one backward pass is enough.
Vanishing gradients, ReLU and other practical issues
The recipe is exact, but the numbers it produces can behave badly in deep networks. Most of the design choices in modern architectures exist to keep gradients a healthy size.
Vanishing and exploding gradients
Look at the recursion δ(l) = (W(l+1))Tδ(l+1) ⊙ σ′(z(l)). Each layer the gradient passes through multiplies it by a weight matrix and by the activation’s derivative. The sigmoid’s derivative is at most 0.25, reached at z = 0, and is tiny when |z| is large.
Ignoring the weights, ten sigmoid layers can shrink a gradient by a factor of up to 0.2510, which is less than one millionth. Early layers then receive almost no signal and barely learn. This is the vanishing gradient problem, and it is a large part of why deep networks were hard to train for many years. You can see a mild version even in our two-layer example: the hidden-layer δs (−0.0206 and 0.0109) are several times smaller than the output δ (−0.0952). The opposite happens when weights are large: the products grow layer after layer, gradients explode, and training becomes unstable. A common guard against that is gradient clipping, which caps the size of the gradient before each update.
ReLU
The rectified linear unit, ReLU(z) = max(0, z), has derivative 1 for positive inputs and 0 for negative ones. Where a unit is active, the gradient passes through untouched instead of being multiplied by at most 0.25. That simple change is one of the main reasons deep networks became trainable, and ReLU and its relatives are now the default hidden-layer activations. The catch is that a unit whose input is always negative gets a gradient of zero and stops learning (a “dead” ReLU), which variants such as leaky ReLU address by giving negative inputs a small slope. The derivative at exactly zero is undefined, and libraries simply pick a value there.
Initialisation
Two things matter. First, weights must start different. If two hidden units begin with identical weights, they compute identical outputs, receive identical gradients and stay identical forever, so random initialisation is essential. Second, the random values need the right scale, so that activations and gradients neither shrink nor grow as they pass through layers. Glorot (also called Xavier) initialisation, with variance 2/(nin + nout), was designed with sigmoid and tanh units in mind; He initialisation, with variance 2/nin, is the usual choice for ReLU. Normalisation layers and residual connections, which give gradients a direct path around blocks of layers, attack the same problem structurally.
Mini-batches
So far the loss came from a single example. The real objective is the average loss over the whole training set, and because differentiation is linear, its gradient is simply the average of the per-example gradients. Computing that exactly for every step means a pass over all the data, which is slow. Using one example per step is cheap but very noisy.
The standard compromise is the mini-batch: estimate the gradient from a small random batch of examples at each step. The matrix form makes this efficient. Stack the batch’s inputs as the columns of a matrix A(0), and every matrix–vector product above becomes a matrix–matrix product that GPUs run in parallel. With m examples, the weight gradient becomes (1/m) Δ(l)(A(l−1))T, which adds up the outer product from each example and averages them. The noise in mini-batch gradients is not purely a cost, either; it is widely thought to help training escape poor regions of the loss surface.
Autograd in practice
Nobody writes backward passes by hand for real models. Frameworks such as PyTorch and JAX record the computational graph as your forward code runs, then apply exactly the node-by-node rules above when you ask for gradients. Here is the article’s network in PyTorch:
import torch
x = torch.tensor([1.0, 0.5])
y = torch.tensor(1.0)
W1 = torch.tensor([[0.5, -0.4],
[0.3, 0.8]], requires_grad=True)
b1 = torch.tensor([0.1, -0.1], requires_grad=True)
W2 = torch.tensor([0.9, -0.5], requires_grad=True)
b2 = torch.tensor(0.2, requires_grad=True)
def loss_fn():
h = torch.sigmoid(W1 @ x + b1) # hidden layer
o = torch.sigmoid(W2 @ h + b2) # output
return 0.5 * (o - y) ** 2
loss = loss_fn()
loss.backward() # backpropagation
print(f"loss {loss.item():.4f}")
print(W1.grad)
print(b1.grad)
print(W2.grad)
print(b2.grad)
with torch.no_grad(): # one gradient-descent step
for p in (W1, b1, W2, b2):
p -= 0.5 * p.grad
print(f"loss after one step {loss_fn().item():.4f}")It prints the same numbers we worked out by hand:
loss 0.0790
tensor([[-0.0206, -0.0103],
[ 0.0109, 0.0054]])
tensor([-0.0206, 0.0109])
tensor([-0.0570, -0.0615])
tensor(-0.0952)
loss after one step 0.0707Row j of W1.grad holds the gradients for the weights into hidden unit j, so the first row is w1, w2 and the second is w3, w4. A few details are worth knowing:
requires_grad=Truemarks the leaves of the graph you want gradients for. The inputs and target do not need it.loss.backward()runs reverse mode from the loss, seeding it with ∂L/∂L = 1, and stores each leaf’s gradient in its.gradattribute.- Gradients accumulate. PyTorch adds into
.gradrather than overwriting it, which is how a value used in several places collects the gradient from each use. The side effect is that a second call tobackward()adds to the old gradients, which is why training loops calloptimizer.zero_grad()before each backward pass. - The update runs inside
torch.no_grad()so that changing the weights is not itself recorded as part of the graph. - For custom operations,
torch.autograd.gradcheckcompares your backward pass against finite differences, the same check used throughout this article.
Summary
- Training needs the gradient of the loss with respect to every parameter. Estimating each one by nudging it costs a forward pass per parameter; backpropagation gets all of them exactly from one forward and one backward pass.
- Draw the computation as a graph of simple nodes. The forward pass computes and stores every intermediate value.
- The chain rule does the rest: multiply local derivatives along a path and add over paths. Going backwards from ∂L/∂L = 1, each node turns the gradient it receives into gradients for its inputs.
- In a network, track δ = ∂L/∂z for each unit. Then ∂L/∂w = δto × afrom and ∂L/∂b = δ. In matrix form, δ(l) = (W(l+1))Tδ(l+1) ⊙ σ′(z(l)).
- Gradient descent uses those gradients: θ ← θ − η∂L/∂θ. In the worked example one step with η = 0.5 cut the loss from 0.0790 to 0.0707.
- A backward pass costs about twice a forward pass, but needs the forward pass’s values kept in memory. It is reverse-mode automatic differentiation, ideal for one scalar loss and many parameters.
- Repeated multiplication makes gradients vanish or explode in deep networks. ReLU activations, careful initialisation, normalisation and residual connections keep them a usable size.
Further reading on this site
- How convolution works in a CNN: convolutional layers are trained with exactly this algorithm, and the backward pass through a convolution turns out to be another convolution.
- How support vector machines work: a different model built around a different loss, the hinge loss, which makes a useful contrast with the smooth losses used here.
- How retrieval-augmented generation (RAG) works: the embedding models behind vector search are neural networks trained with backpropagation.