Mode Collapse & GAN Instability

Why GAN Training is Notoriously Unstable

Training a standard deep neural network minimizes a fixed loss function via gradient descent down a stable bowl.

Training a GAN involves two competing networks playing a Minimax Game. Instead of descending a bowl, the system attempts to find a Nash Equilibrium in a non-convex vector space.

This game dynamics creates three major training failure modes:

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. MODE COLLAPSE         │ 2. VANISHING GRADIENTS   │ 3. UNBALANCED COMPETITION│
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Generator finds ONE fake │ Discriminator becomes    │ If D is too strong, G    │
│ output that tricks D and │ perfect instantly. JS    │ learns nothing. If G is  │
│ outputs it repeatedly,   │ divergence derivative    │ too strong, D outputs    │
│ ignoring data diversity. │ vanishes to zero!        │ random noise.            │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Mode Collapse

Imagine training a GAN on 10 digits (0 through 9).

The Generator discovers that outputting a realistic digit 8 tricks the Discriminator 90 percent of the time.

Instead of learning digits 0 through 9, the Generator collapses into generating only digit 8 repeatedly!

  Expected Generator Output:  [ Digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ]
  Mode Collapse Output:       [ Digit 8, Digit 8, Digit 8, Digit 8 ]  <-- Lost Data Diversity!

2. Vanishing Gradients & JS Divergence

Standard GAN loss implicitly minimizes the Jensen-Shannon (JS) Divergence between real distribution $P_r$ and fake distribution $P_g$.

If $P_r$ and $P_g$ reside on low-dimensional manifolds with zero spatial overlap:

$$\text{JS}(P_r \parallel P_g) = \log(2) \quad (\text{Constant!})$$

Because JS divergence is constant $\log(2)$ when distributions do not overlap, its derivative is ZERO. The Discriminator easily rejects fake images, but provides zero gradient signal back to the Generator!

3. The Solution: Wasserstein GAN (WGAN & WGAN-GP)

Arjovsky et al. (2017) replaced JS divergence with Wasserstein-1 Distance (Earth Mover's Distance):

$$W(P_r, P_g) = \inf_{\gamma \in \Pi(P_r, P_g)} \mathbb{E}_{(x, y) \sim \gamma} [|x - y|]$$

Wasserstein distance measures the minimum work required to transport probability mass from $P_g$ to match $P_r$.

  WASSERSTEIN ADVANTAGE:
  - Continuous and differentiable EVERYWHERE, even when distributions have zero overlap!
  - Provides smooth, non-saturating gradients back to the Generator throughout training.
  - Completely eliminates Mode Collapse!

Enforcing 1-Lipschitz Continuity (WGAN-GP)

To compute Wasserstein distance, the Discriminator (now called a Critic) must satisfy 1-Lipschitz Continuity ($|\nabla_x D(x)| \le 1$).

WGAN-GP (Gulrajani et al., 2017) enforces this by adding an explicit Gradient Penalty term directly to the loss function:

$$\mathcal{L}{\text{WGAN-GP}} = \mathbb{E}[D(\hat{x})] - \mathbb{E}[D(x)] + \lambda \mathbb{E} \left[ (|\nabla{\hat{x}} D(\hat{x})|_2 - 1)^2 \right]$$

Say this out loud

GAN training instability manifests as Mode Collapse, where the Generator outputs a single repetitive image, and Vanishing Gradients when the Discriminator becomes too dominant. Wasserstein GAN (WGAN-GP) replaces JS divergence with Earth Mover Distance and Gradient Penalties, providing smooth continuous gradients everywhere and eliminating Mode Collapse.

Followups to expect

1. What is Spectral Normalization (Miyato et al., 2018)? Normalizing weight matrices by their largest singular value (spectral norm) to enforce 1-Lipschitz continuity in Discriminator layers, offering fast, stable GAN training.
2. What is Frechet Inception Distance (FID)? The standard evaluation metric for GAN image quality. FID compares mean and covariance feature distributions of real vs generated images extracted from an Inception-v3 model. Lower FID indicates higher quality.