CNN vs ViT: Inductive Bias
Comparing local inductive bias in ConvNets against global self-attention scaling in Vision Transformers.
Structural Comparison
CONVOLUTIONAL NEURAL NETWORK (CNN) VISION TRANSFORMER (ViT)
Local 3x3 Sliding Window Receptive Fields Global 16x16 Patch Token Self-Attention
┌───┬───┬───┐ ┌──────┬──────┐
│ . │ . │ . │ Local Locality │Patch1│Patch2│ Global Interactions
├───┼───┼───┤ Hard-Coded Inductive Bias! ├──────┼──────┤ Learned Spatial Relationships!
│ . │ . │ . │ │Patch3│Patch4│
└───┴───┴───┘ └──────┴──────┘
Detailed Comparison Matrix
| Property | Convolutional Neural Network (ResNet / ConvNeXt) | Vision Transformer (ViT / Swin) |
|---|---|---|
| Core Operation | Local 3x3 / 7x7 Sliding Convolutions | Global / Windowed Multi-Head Self-Attention |
| Inductive Bias | High (Local Locality & Translation Invariance) | Minimal (Learns spatial layout from data) |
| Small Dataset Performance | Strong (Works well on ImageNet-1K from scratch) | Poor without heavy augmentation / pre-training |
| Large Scale Data Scaling | Saturates earlier | Scales exceptionally with compute & data |
| Receptive Field Growth | Linear with depth ($O(L)$ layers) | Global immediately in Layer 1 ($O(1)$) |
| Computational Complexity | $O(H \cdot W \cdot C^2)$ (Linear with resolution) | $O((HW)^2 \cdot C)$ (Quadratic with resolution) |
Patch Tokenization Math in ViT
Given input image $X \in \mathbb{R}^{H \times W \times C}$ and patch size $P = 16$:
- Number of patch tokens $N$:
$$N = \frac{H \cdot W}{P^2}$$
- Each patch is flattened into vector $x_p \in \mathbb{R}^{P^2 \cdot C}$.
- Linear projection $E \in \mathbb{R}^{(P^2 \cdot C) \times D}$ maps patches to $D$-dimensional embeddings.
- Prepend learnable
[CLS]token and add 1D Learnable Position Embeddings $E_{\text{pos}}$:
$$\mathbf{z}0 = [ x{\text{class}}; x_p^1 E; x_p^2 E; \dots; x_p^N E ] + E_{\text{pos}}$$
- Pass $\mathbf{z}_0$ into standard Transformer Encoder layers!
Hybrid & Swin Transformers
Standard ViT self-attention has quadratic $O(N^2)$ complexity with respect to image pixels, making high-resolution processing slow.
- Swin Transformer (Liu et al., 2021): Computes self-attention within Shifted Local Windows, achieving linear $O(N)$ complexity and hierarchical multi-scale feature pyramids for object detection and segmentation.
Say this out loud
CNNs use local sliding convolutions with strong spatial inductive biases (locality and translation invariance), excelling on smaller datasets. Vision Transformers (ViT) tokenize images into 16x16 patch tokens and use global self-attention. Lacking inductive bias, ViTs require massive pre-training data (JFT-300M) or Swin shifted windows, but scale significantly better with compute.
Follow-ups to expect
- What is Masked Autoencoder (MAE - He et al., 2021)? Self-supervised ViT pre-training technique that masks 75% of input image patches and trains an autoencoder to reconstruct missing pixels, achieving state-of-the-art vision representations.
- What is Attention Map Visualization in ViT? Inspecting self-attention weights from the
[CLS]token to patch tokens reveals that ViT naturally attends to semantic object boundaries without explicit bounding box supervision.
Check yourself
What spatial Inductive Biases are hard-coded into Convolutional Neural Networks (CNNs) that Vision Transformers (ViT) lack?