ResNet, EfficientNet & Friends

Evolution of Image Classification Architectures

┌──────────────────────────┬──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. ALEXNET (2012)        │ 2. VGG (2014)            │ 3. RESNET (2015)         │ 4. CONVNEXT (2022)       │
├──────────────────────────┼──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Sparked Deep Learning    │ Uniform 3x3 convolutions.│ Residual skip shortcuts: │ Modernized CNN with ViT  │
│ era. Used ReLU, Dropout, │ Deep stacked layers      │ y = F(x) + x. Scaled to  │ tricks: 7x7 depthwise    │
│ GPU training (ImageNet). │ (VGG-16, VGG-19).        │ 152+ layers safely.      │ blocks & LayerNorm.      │
└──────────────────────────┴──────────────────────────┴──────────────────────────┴──────────────────────────┘

The Degradation Problem & ResNet Skip Connections

Before ResNet, stacking more layers caused training accuracy to saturate and degrade rapidly.

This was not caused by overfitting, but by vanishing gradients: signals decayed exponentially through 50+ stacked matrix multiplications.

  TRADITIONAL LAYER:        Input x ──► [ Conv Block F(x) ] ──────────────► Output F(x)
                                                                           (Gradients decay!)

  RESIDUAL BLOCK (ResNet):  Input x ──┬─► [ Conv Block F(x) ] ──► (+) ────► Output F(x) + x
                                      │                            ▲       (Identity Path!)
                                      └───── Identity Shortcut ────┘

The residual block forces the network to learn the residual mapping $F(x) = H(x) - x$ rather than fitting raw mapping $H(x)$ from scratch.

Because gradient $\frac{\partial (F(x) + x)}{\partial x} = \frac{\partial F}{\partial x} + 1$, gradients always have a clean $+1$ constant path to flow backward to early layers!

VGG Factorization: Why 3x3 Convolutions Win

• Stacking two 3x3 conv layers gives an effective receptive field of 5x5.
• Stacking three 3x3 conv layers gives an effective receptive field of 7x7.

Parameter savings for $C$ channels:

Three 3x3 layers: $3 \times (3^2 \cdot C^2) = 27 C^2$ parameters.

One 7x7 layer: $1 \times (7^2 \cdot C^2) = 49 C^2$ parameters.

Stacking smaller 3x3 filters saves 45% parameters while introducing 3 non-linear ReLU activations instead of 1!

ConvNeXt: CNN Strikes Back

ConvNeXt took a standard ResNet-50 and systematically updated its design using Vision Transformer (Swin) techniques:

1. Depthwise Separable Convolutions: Separates spatial filtering from channel mixing.
2. Larger Kernel Size: Upgraded from 3x3 to 7x7 kernels.
3. Inverted Bottleneck: 1x1 conv expands channels by 4x, followed by 7x7 depthwise conv, then 1x1 conv shrinks channels.
4. Modern Micro-Design: Replaced ReLU with GELU, BatchNorm with LayerNorm, and reduced normalization frequency.

Say this out loud

Computer vision architectures evolved from AlexNet to ResNet and ConvNeXt. ResNet solved vanishing gradients in deep 100+ layer networks by adding residual identity skip connections y = F(x) + x, providing an un-attenuated gradient shortcut during backpropagation. ConvNeXt modernized CNNs by incorporating 7x7 depthwise convolutions and LayerNorm, matching Vision Transformer accuracy while preserving fast CNN inference.

Follow-ups to expect

• What is a Bottleneck Block in ResNet-50/101/152? Uses 1x1 conv to reduce dimensions (e.g. 256 -> 64), 3x3 conv to process features, and 1x1 conv to restore dimensions (64 -> 256), cutting FLOPs by 4x compared to basic blocks.
• What is MobileNet? Efficient CNN architecture for mobile devices that replaces standard convolutions with Depthwise Separable Convolutions (Depthwise Conv + Pointwise 1x1 Conv), reducing FLOPs and parameter count by 8x-9x.