Vision Transformers (ViT)

An Image is Worth 16x16 Words

Before 2020, Convolutional Neural Networks (CNNs) were considered mandatory for computer vision due to their built-in spatial inductive biases (locality and translation invariance).

Vision Transformers (ViT - Dosovitskiy et al., 2020 / Google) proved that standard Transformer architectures could achieve state-of-the-art vision accuracy with zero convolutional layers!

  Input Image (224x224x3) ──► Split into 16x16 Patches (196 Patches!) ──► Linear Patch Projection
                                                                                  │
                                                                                  ▼
  [ [CLS] Token + 196 Patch Tokens ] + Positional Encodings ──► [ TRANSFORMER ENCODER STACK ]
                                                                                  │
                                                                                  ▼
                                                            Linear MLP Head on [CLS] Output ──► Class!

How ViT Works Step-by-Step

Given an image of shape $H \times W \times C$ (e.g. $224 \times 224 \times 3$):

Step 1: Patch Extraction & Flattening

Split the image into non-overlapping patches of size $P \times P$ (typically $P = 16$).

Total number of patches $N$:

$$N = \frac{H \cdot W}{P^2} = \frac{224 \cdot 224}{16^2} = 14 \cdot 14 = 196 \text{ Patches}$$

Flatten each 2D patch into a 1D vector of length $P^2 \cdot C = 16 \times 16 \times 3 = 768$.

Step 2: Linear Projection & Positional Embedding

Multiply flattened patch vectors by a projection matrix $E \in \mathbb{R}^{768 \times d_{\text{model}}}$ to map them to model dimension $d_{\text{model}}$.

Prepend a learnable Classification Token [CLS] and add 1D Learnable Positional Embeddings to retain spatial location order.

Step 3: Transformer Encoder & MLP Head

Pass the 197 token sequence ([CLS] + 196 patches) through standard Transformer Encoder blocks (Multi-Head Self Attention + FFN).

The final output vector corresponding to the [CLS] token is passed to a Linear MLP Head to output class probabilities.

CNNs vs ViTs: Inductive Bias & Scaling

┌──────────────────────────┬──────────────────────────┐
│ CONVOLUTIONAL NETS (CNN) │ VISION TRANSFORMERS (ViT)│
├──────────────────────────┼──────────────────────────┤
│ Strong Spatial Bias      │ Weak Inductive Bias      │
│ (Locality + Translation) │ (Must learn spatial      │
│ High accuracy on SMALL   │ relationships from data) │
│ datasets (ImageNet-1k).  │ Underperforms on small!  │
│ Slower scaling on compute│ Outstanding scaling on   │
│                          │ MASSIVE data (JFT-300M)! │
└──────────────────────────┴──────────────────────────┘

When trained from scratch on small datasets (ImageNet-1k), ViT underperforms ResNet because it lacks spatial assumptions.

When pretrained on massive datasets (JFT-300M or ImageNet-21k), ViT outperforms CNNs, scaling smoothly with larger compute budgets.

Say this out loud

Vision Transformers split images into grids of non-overlapping 16x16 pixel patches, flattening them into 1D token sequences for a standard Transformer Encoder. While ViTs lack convolutional spatial inductive biases and require massive pretraining datasets like JFT 300M, they scale better than CNNs on large compute budgets.

Followups to expect

1. What is Swin Transformer (Liu et al., 2021)? Introduces Shifted Windows to compute self attention locally within non overlapping image windows, reducing computational complexity from quadratic $O(N^2)$ to linear $O(N)$ for high resolution vision tasks.
2. What is Masked Autoencoder (MAE - He et al., 2022)? A self supervised pretraining task for ViT that masks out 75 percent of image patches, training the model to reconstruct raw pixel patches from the remaining 25 percent visible patches.