How does backpropagation train a neural network?

Question

AcadiFi · Accepted Answer

Backpropagation computes the gradient of a loss function with respect to every weight in the network, using the chain rule. The training loop:

1. **Forward pass**: compute predictions for a batch.
2. **Loss computation**: compare predictions with labels (e.g., mean squared error or cross-entropy).
3. **Backward pass**: propagate error gradients from output back to input.
4. **Weight update**: use gradient descent, w_new = w_old - eta x dL/dw.
5. Repeat until convergence.

Core formula: for layer l, the error signal delta^l = f'(z^l) * (W^{l+1})^T delta^{l+1}, where * is element-wise multiplication. Gradient of any weight wij^l = delta_i^l x a_j^{l-1}.

Example: 3-2-1 architecture with sigmoid activations. Inputs [1.2, -0.8, 0.5], target y=0.7. Forward pass produces prediction y_hat=0.55, loss L = 0.5(0.55 - 0.7)^2 = 0.01125.

Output layer gradient: dL/dz_out = (y_hat - y) x f'(z_out). For sigmoid, f' = f(1-f) = 0.55 x 0.45 = 0.2475. dL/dz_out = -0.15 x 0.2475 = -0.0371. Gradients of output weights = delta_out x hidden_activations.

Propagate back: hidden deltas = f'(z_hidden) x W_out x delta_out. Update weights with eta = 0.05. After many batches, weights align to minimize the loss surface.

Key challenges:
- **Vanishing gradients**: sigmoid f' is at most 0.25, so gradients shrink geometrically in deep networks. ReLU fixes this (f'=1 for positive inputs).
- **Exploding gradients**: large weights or poorly conditioned inputs can blow up deltas. Gradient clipping helps.
- **Local minima and saddle points**: SGD with momentum or Adam optimizer helps escape.

Variants: Adam optimizer adapts per-parameter learning rates using first- and second-moment estimates of gradients.

How does backpropagation train a neural network?

Master Level II with our CFA Course

Related Questions

Practice Questions