Entropy, Cross-Entropy & KL Divergence

The Trilogy Defined

Given true distribution P and predicted distribution Q:

1. Shannon Entropy H(P)

Measures average information (surprise / uncertainty) in P:

H(P) = - ∑ P(x) log P(x)

• Deterministic event (P=1): H(P) = 0 (Zero surprise).
• Uniform distribution (P=1/K): Maximum entropy log(K) (Maximum uncertainty).

2. Cross-Entropy H(P, Q)

Expected cost of encoding events from true P using model Q:

H(P, Q) = - ∑ P(x) log Q(x)

In classification, for one-hot label P where P(y=c)=1 and P(y≠c)=0: H(P, Q) = - log Q(y=c).

3. KL Divergence D_KL(P || Q)

Extra bits needed when encoding P using Q (relative entropy):

D_KL(P || Q) = ∑ P(x) log( P(x) / Q(x) )  =  H(P, Q) - H(P)

   Cross-Entropy H(P, Q)  =  True Entropy H(P)  +  KL Divergence D_KL(P || Q)
  (What loss optimizes)       (Fixed constant)       (Distance to truth)

Key Properties of KL Divergence

• Non-negativity (Gibbs' Inequality): D_KL(P || Q) ≥ 0, with equality iff P = Q.
• Asymmetry: D_KL(P || Q) ≠ D_KL(Q || P).
  • Forward KL D_KL(P || Q) (Mean-seeking): Forces Q to cover all modes of P (used in Maximum Likelihood).
  • Reverse KL D_KL(Q || P) (Mode-seeking): Forces Q to fit tightly into a single mode of P (used in Variational Inference / VAEs).

Applications in Deep Learning

1. Classification Loss: Minimizing Cross-Entropy loss pushes model distribution Q toward true target distribution P.
2. Variational Autoencoders (VAEs): Loss combines reconstruction MSE with KL Divergence D_KL( q(z|x) || N(0,I) ) to regularize latent space.
3. RLHF / PPO in LLMs: Adds KL penalty D_KL( π_RL || π_SFT ) to prevent fine-tuned policy LLM from drifting too far from base SFT model.

Say this out loud

"Entropy H(P) measures inherent uncertainty in a distribution. Cross-Entropy H(P,Q) measures the surprise of observing true distribution P under model distribution Q. KL Divergence D_KL(P||Q) = H(P,Q) - H(P) measures the non-symmetric information loss when approximating P with Q. Because H(P) is constant for fixed ground-truth labels, minimizing Cross-Entropy loss is mathematically identical to minimizing KL Divergence."

Follow-ups to expect

• What is Jensen-Shannon (JS) Divergence? A symmetric, bounded version of KL Divergence: JS(P||Q) = 1/2 D_KL(P||M) + 1/2 D_KL(Q||M), where M = 1/2(P + Q). Used as the original theoretical loss metric in Generative Adversarial Networks (GANs).
• Why is Cross-Entropy preferred over MSE for classification logits? MSE with Sigmoid/Softmax has flat saturated regions where gradients vanish. Cross-Entropy yields linear gradients (p - y) that accelerate convergence.