Mutual Information

What is Mutual Information?

Mutual Information $I(X; Y)$ measures how much knowing variable $Y$ reduces uncertainty about variable $X$.

                       INFORMATION THEORY VENN DIAGRAM
                     ┌───────────────────┬───────────────────┐
                     │   Entropy H(X)    │   Entropy H(Y)    │
                     │                   │                   │
                     │         ┌─────────┴─────────┐         │
                     │         │ MUTUAL INFO I(X;Y)│         │
                     │ H(X|Y)  │  Joint overlap    │ H(Y|X)  │
                     │         │                   │         │
                     └─────────┴───────────────────┴─────────┘

Three Equivalent Formulations

1. Entropy Reduction:

$$I(X; Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X)$$

2. Symmetric Joint Overlap:

$$I(X; Y) = H(X) + H(Y) - H(X, Y)$$

3. KL Divergence from Independence:

$$I(X; Y) = D_{KL}\left( P(X, Y) \parallel P(X) P(Y) \right) = \sum_{x \in X} \sum_{y \in Y} P(x, y) \log \frac{P(x, y)}{P(x) P(y)}$$

• If $X$ and $Y$ are independent $\implies P(X, Y) = P(X)P(Y) \implies I(X; Y) = 0$.
• If $X$ is a deterministic function of $Y \implies I(X; Y) = H(X)$.

Pearson Correlation vs Mutual Information

Consider data generated by $y = x^2$ for $x \sim \text{Uniform}(-1, 1)$:

  y = x^2 (Parabola)
  y
  1 ┤  *       *
    │   *     *
  0 ┴────*──*──*────► x
       -1   0  1

• Pearson Correlation ($r$): $r = 0.0$! Pearson concludes $X$ and $Y$ have zero relationship because the relationship is non-linear.
• Mutual Information ($I(X;Y)$): $I(X; Y) > 0$. Correctly detects strong statistical dependence!

Applications in Machine Learning

1. Non-Linear Feature Selection: Rank features $X_i$ by $I(X_i; Y)$ with target variable $Y$, filtering out irrelevant features regardless of model architecture.
2. Contrastive Representation Learning (InfoNCE): Self-supervised models (SimCLR, CLIP) maximize InfoNCE loss:

$$\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E} \left[ \log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_k \exp(\text{sim}(z_i, z_k) / \tau)} \right]$$

InfoNCE maximizes a lower bound on Mutual Information $I(z_i; z_j)$ between augmented views of the same image!

Say this out loud

Mutual Information I(X;Y) measures how much knowing Y reduces uncertainty about X, calculated as H(X) - H(X|Y) or KL divergence between joint P(X,Y) and independent product P(X)P(Y). Unlike Pearson correlation which finds only linear patterns, Mutual Information detects arbitrary non-linear dependencies, underpinning non-linear feature selection and contrastive InfoNCE representation learning.

Follow-ups to expect

• What is Data Processing Inequality (DPI)? If $X \to Y \to Z$ forms a Markov chain, then $I(X; Y) \ge I(X; Z)$. Post-processing data $Y$ cannot increase information about original variable $X$.
• What is Disentangled Representation Learning (beta-VAE)? Modifies VAE loss to minimize mutual information between latent code variables, forcing individual latent dimensions to capture independent physical factors (e.g. skin color, rotation angle).