Variational Autoencoders

What is a Variational Autoencoder (VAE)?

A standard autoencoder compresses input $x$ into a fixed point vector $z$.

If you try to generate a new image by sampling a random vector $z$, the decoder outputs garbled noise because standard latent space is full of empty unmapped gaps.

A Variational Autoencoder (VAE) maps inputs to a Continuous Probability Distribution (mean $\mu$ and variance $\sigma^2$) in latent space:

                               VAE LATENT PROBABILISTIC SPACE
  Input x ──► [ ENCODER ] ──┬──► Mean Vector mu [1 x D]       ──┐
                            └──► Variance Vector sigma [1 x D] ─┼──► Sample z = mu + sigma * eps ──► [ DECODER ] ──► x_hat
                                                                │    (Reparameterization Trick!)
                                                                ▼
                                                   Unit Normal eps ~ N(0, I)

The VAE Loss Function: Evidence Lower Bound (ELBO)

VAEs optimize two objectives simultaneously:

$$\text{Loss}_{\text{VAE}} = \text{Reconstruction Loss} + \text{KL Divergence Loss}$$

$$\mathcal{L}{\text{ELBO}} = \mathbb{E}{q(z|x)} [ \log p(x|z) ] - D_{KL}\left( q(z|x) \parallel \mathcal{N}(0, I) \right)$$

1. Reconstruction Loss: Measures how accurately the decoder reconstructs the input image (MSE or Cross Entropy).
2. KL Divergence Loss: Regularizes the learned latent distribution $q(z|x)$ to stay close to a standard Unit Gaussian distribution $\mathcal{N}(0, I)$.

┌──────────────────────────┬──────────────────────────┐
│ RECONSTRUCTION LOSS      │ KL DIVERGENCE LOSS       │
├──────────────────────────┼──────────────────────────┤
│ Forces distinct inputs to│ Forces all distributions │
│ map to separate regions. │ to overlap around 0.     │
│ Prevents memory loss.    │ Creates smooth space     │
│                          │ without empty gaps!      │
└──────────────────────────┴──────────────────────────┘

The competition between these two loss terms creates a smooth, continuous, densely packed latent space where interpolating between two latent points smoothly morphs one image into another!

The Reparameterization Trick

To train a VAE, we must sample latent vector $z \sim \mathcal{N}(\mu, \sigma^2)$.

However, drawing a random sample is a non differentiable stochastic operation, which blocks backpropagation gradients from flowing back into the Encoder.

The Reparameterization Trick decouples the stochastic randomness from trainable parameters:

$$z = \mu + \sigma \odot \epsilon, \quad \text{where } \epsilon \sim \mathcal{N}(0, I)$$

1. $\epsilon$ is drawn independently from a fixed Unit Gaussian $\mathcal{N}(0, I)$ (Zero trainable parameters!).
2. Gradients for $\mu$ and $\sigma$ flow smoothly backward through standard multiplication and addition operations!

Say this out loud

A Variational Autoencoder maps inputs to mean and variance vectors of a Gaussian latent distribution. The ELBO loss combines Reconstruction Loss with KL Divergence regularization to force latent distributions to overlap around a unit Gaussian, creating a continuous gap-free latent space for generative sampling. The Reparameterization Trick z = mu + sigma * epsilon enables backpropagation through stochastic random sampling.

Followups to expect

1. What is Latent Space Interpolation in VAEs? Taking two latent vectors z1 and z2, stepping linearly between them (z = alpha * z1 + (1 - alpha) * z2), and passing intermediate vectors through the decoder to smoothly morph image features (like morphing a smiling face into a neutral face).
2. Why do VAE generated images look slightly blurry compared to GANs? VAEs minimize mean squared reconstruction loss, which averages pixel uncertainties, producing smooth blurry outputs. GANs use adversarial discriminators that penalize blurriness directly.