Residual Connections

The Deep Network Degradation Problem

Before ResNet in 2015, conventional wisdom held that adding more layers to a neural network would always improve learning capacity.

However, researchers discovered an unexpected problem: stacking more than 20 or 30 standard layers caused training error to saturate and degrade rapidly.

This was not overfitting. Training loss itself became worse on 56 layer plain networks than on 20 layer networks!

Repeated matrix multiplications caused gradients to decay exponentially to near zero as they traveled backward, preventing early layers from updating.

The Residual Solution: Identity Shortcuts

ResNet (He et al., 2015) introduced a simple architectural modification called a Residual Connection:

y = F(x) + x

Instead of forcing layers to fit a complex target mapping $H(x)$ from scratch, the residual block forces layers to fit a small residual delta $F(x) = H(x) - x$.

                       RESIDUAL BLOCK ARCHITECTURE
                       Input Tensor x
                             │
            ┌────────────────┴────────────────┐
            │                                 │
            ▼                                 │ Identity Shortcut
   [ Layer 1 (Conv 3x3) ]                     │ (Zero Parameters!)
            │                                 │
            ▼                                 │
   [ Layer 2 (Conv 3x3) ]                     │
            │                                 │
            ▼                                 │
           (+) ◄──────────────────────────────┘
            │
            ▼
     Output y = F(x) + x

Why Residual Connections Solve Vanishing Gradients

During backpropagation, we calculate the derivative of output $y$ with respect to input $x$:

$$\frac{\partial y}{\partial x} = \frac{\partial F(x)}{\partial x} + 1$$

Using the chain rule for loss $L$:

$$\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \left( \frac{\partial F(x)}{\partial x} + 1 \right) = \frac{\partial L}{\partial y} \cdot \frac{\partial F(x)}{\partial x} + \frac{\partial L}{\partial y}$$

Notice the $+ \frac{\partial L}{\partial y}$ term!

Even if the sub layer derivative $\frac{\partial F}{\partial x}$ decays to zero, the gradient $\frac{\partial L}{\partial y}$ still flows backward completely unattenuated through the $+ 1$ identity shortcut!

This creates a high speed gradient highway directly connecting the final loss to the very first input layers.

Residual Connections in Modern AI

Residual connections are mandatory in almost all state of the art deep learning architectures:

1. Computer Vision: ResNet, ConvNeXt, EfficientNet.
2. Transformers: BERT, GPT 4, LLaMA 3, Claude 3 (LayerNorm + SubLayer(x) + x).
3. Diffusion Models: U-Net Denoising Backbones in Stable Diffusion.

Say this out loud

Residual connections add the un-transformed input tensor directly to a block output y = F(x) + x. During backpropagation, the derivative of the identity path is 1.0, creating an unattenuated gradient highway back to early layers. This solved vanishing gradients, allowing networks to scale past 100 layers in ResNet and forming a core building block of modern Transformers.

Followups to expect

1. What is Pre LayerNorm vs Post LayerNorm in Transformers? Post LayerNorm applies normalization after the residual addition (x + SubLayer(x)). Pre LayerNorm applies normalization before the sub layer (x + SubLayer(Norm(x))), offering superior gradient stability for training deep LLMs.
2. What happens if input x and output F(x) have different spatial dimensions? Apply a 1x1 linear projection convolution to input x along the shortcut path so tensor shapes match before addition.