Backpropagation

Why reverse-mode automatic differentiation?

Consider a function L = f(x) with M inputs and N outputs.

• Forward-mode AD: Computes ∂y / ∂x_i by tracking directional derivatives forward. Takes O(M) sweeps.
• Reverse-mode AD (Backprop): Computes ∂L_j / ∂x by tracking adjoint signals backwards from loss. Takes O(N) sweeps.

In neural networks, weights M = 10⁶ to 10¹¹, while scalar loss N = 1. Backpropagation evaluates gradients wrt all parameters in a single reverse sweep taking O(W) computation.

The Chain Rule & Error Vector (δ)

For layer l:

• Pre-activation: z^(l) = W^(l) a^(l-1) + b^(l)
• Activation: a^(l) = σ( z^(l) )

Define error signal δ^(l) = ∂L / ∂z^(l).

Using the chain rule:

δ^(L) = ∂L / ∂a^(L) ⊙ σ'(z^(L))                  (Output layer error)

δ^(l) = ( (W^(l+1))ᵀ δ^(l+1) ) ⊙ σ'(z^(l))        (Backpropagated error signal)

Gradients with respect to parameters at layer l:

∂L / ∂W^(l) = δ^(l) (a^(l-1))ᵀ
∂L / ∂b^(l) = δ^(l)

Each layer takes incoming gradient δ^(l+1), computes local gradient σ'(z), passes δ^(l) to previous layer, and accumulates parameter gradients.

Matrix calculus dimension sanity check

In interviews, never guess matrix derivatives — use shape matching:

Given y = X W + b where:

• X is batch input: [Batch, Input_Dim]
• W is weight matrix: [Input_Dim, Output_Dim]
• y is layer output: [Batch, Output_Dim]
• dL/dy is incoming upstream gradient: [Batch, Output_Dim]

Target shapes for gradients:

• dL/dW must be [Input_Dim, Output_Dim] → dL/dW = Xᵀ (dL/dy) ([Input_Dim, Batch] × [Batch, Output_Dim])
• dL/dX must be [Batch, Input_Dim] → dL/dX = (dL/dy) Wᵀ ([Batch, Output_Dim] × [Output_Dim, Input_Dim])

Say this out loud

"Backpropagation is reverse-mode automatic differentiation applied to computational graphs. Because neural nets map millions of weights to a single scalar loss, reverse mode computes all weight gradients in O(1) backward pass rather than O(W) forward sweeps. It uses the chain rule to recursively multiply local derivatives with upstream gradients, storing forward activations in memory to compute weight updates."

Follow-ups to expect

• Why do vanishing/exploding gradients occur in deep networks during backprop? Repeated matrix multiplication by weight matrices Wᵀ and derivative terms σ'(z) across L layers leads to exponential growth or decay (λ^L) depending on singular values.
• What is gradient checkpointing? A memory optimization that trades compute for memory: instead of caching all intermediate activations during forward pass, it saves activations at key checkpoint layers and recomputes missing intermediate activations on-demand during backward pass.
• Why call .zero_grad() in PyTorch before loss.backward()? PyTorch accumulates (sums) gradients into .grad tensors by default to support features like gradient accumulation over mini-batches.