Generative Adversarial Networks

What is a Generative Adversarial Network (GAN)?

Invented by Ian Goodfellow et al. in 2014, Generative Adversarial Networks (GANs) revolutionized generative modeling.

Instead of maximizing data likelihood, GANs frame generative learning as a Two Player Zero Sum Minimax Game between two competing neural networks:

  Random Noise Vector z ──► [ GENERATOR G ] ──► Fake Image G(z) ──┐
                                                                   ├──► [ DISCRIMINATOR D ] ──► Real or Fake?
  Real Training Dataset ──────────────────────► Real Image x    ──┘

1. The Generator ($G$): Takes a random noise vector $z \sim p_z(z)$ and attempts to synthesize realistic data samples $G(z)$ to fool the Discriminator.
2. The Discriminator ($D$): Acts as an Inspector, taking images and predicting probability $D(x) \in [0, 1]$ that the sample is Real ($1$) or Fake ($0$).

The Minimax Game Equation

$$\min_G \max_D V(D, G) = \mathbb{E}{x \sim p{\text{data}}(x)} [\log D(x)] + \mathbb{E}_{z \sim p_z(z)} [\log(1 - D(G(z)))]$$

• Discriminator Goal ($\max_D$): Wants $D(x) \to 1$ for real images and $D(G(z)) \to 0$ for fake images.
• Generator Goal ($\min_G$): Wants $D(G(z)) \to 1$, fooling the Discriminator into classifying fake images as real.

┌──────────────────────────┬──────────────────────────┐
│ DISCRIMINATOR STEP       │ GENERATOR STEP           │
├──────────────────────────┼──────────────────────────┤
│ Maximize Log Likelihood  │ Minimize Log Likelihood  │
│ of correctly classifying │ of Discriminator spot    │
│ Real vs Fake images.     │ ting fake images.        │
└──────────────────────────┴──────────────────────────┘

Nash Equilibrium

As training progresses:

1. Generator learns to produce increasingly realistic images.
2. Discriminator becomes increasingly sharp at catching subtle fake details.

Training reaches completion at Nash Equilibrium:

$$p_g = p_{\text{data}} \implies D(x) = 0.5$$

The Generator produces synthetic samples so realistic that the Discriminator can no longer tell real from fake, outputting random guessing probability 0.5 for all samples!

Deep Convolutional GAN (DCGAN - Radford et al., 2015)

Original GANs used fully connected dense layers, producing noisy small images.

DCGAN introduced architectural guidelines for Stable Vision GANs:

1. Replace spatial pooling with Strided Convolutions in Discriminator and Fractionally-Strided (Transposed) Convolutions in Generator.
2. Use Batch Normalization in both Generator and Discriminator.
3. Use LeakyReLU activations in Discriminator for all layers.

Say this out loud

GANs frame generative learning as a two player zero sum game between a Generator synthesizing fake samples and a Discriminator classifying real vs fake. Training minimizes the minimax objective until reaching Nash Equilibrium where fake samples match real data distributions and D(x) outputs 0.5 random guess probability.

Followups to expect

1. What is Non-Saturating GAN Loss? Early in training when G is weak, $D(G(z)) \approx 0$, causing $\log(1 - D(G(z)))$ gradients to saturate and vanish. In practice, Generator maximizes $\log D(G(z))$ instead to keep strong early gradients.
2. What are Conditional GANs (cGAN)? Feeding class labels $c$ into both Generator $G(z, c)$ and Discriminator $D(x, c)$ to generate specific target categories (like generating a specific digit or animal).