Perplexity

Mathematical Definition

For a sequence of $N$ tokens $W = (w_1, w_2, \dots, w_N)$:

$$\text{Perplexity}(W) = P(w_1, w_2, \dots, w_N)^{-\frac{1}{N}} = \exp\left( -\frac{1}{N} \sum_{i=1}^N \ln P(w_i \mid w_{<i}) \right)$$

Recognizing Average Cross-Entropy Loss $\mathcal{L} = -\frac{1}{N} \sum \ln P(w_i \mid w_{<i})$:

$$\text{Perplexity} = e^{\mathcal{L}}$$

  Cross-Entropy Loss (L)     Perplexity (PPL = e^L)     Interpretation
  ─────────────────────────────────────────────────────────────────────────────────────────────
  L = 0.0                    PPL = 1.0                  100% Perfect Certainty (0 uncertainty)
  L = 1.61                   PPL = 5.0                  Choosing among 5 equally likely tokens
  L = 2.30                   PPL = 10.0                 Choosing among 10 equally likely tokens
  L = 6.90                   PPL = 1000.0               High Uncertainty / Poor Model

Intuitive Interpretation: Effective Branching Factor

Imagine guessing the next word in a sentence:

• Uniform Guessing over 50,000 Vocab: $\text{PPL} = 50,000$.
• Trained LLaMA-3 Model: $\text{PPL} = 8.2$.

A Perplexity of $8.2$ means the trained model has narrowed down the effective choices at each token step to ~8 weighted candidates.

Key Pitfalls in Perplexity Evaluation

1. Tokenizer Dependency: Comparing LLaMA (32k vocab) vs GPT-4 (100k vocab) using raw per-token PPL is invalid. Use Bits Per Byte (BPB):

$$\text{BPB} = \frac{\mathcal{L}{\text{token}} \cdot N{\text{tokens}}}{\ln(2) \cdot N_{\text{bytes}}}$$

2. Perplexity $\neq$ Alignment / Factuality: A model can achieve ultra-low perplexity by outputting perfectly grammatical, highly fluent text that is completely false or toxic. Perplexity measures fluency/predictability, not truth.

Say this out loud

"Perplexity (PPL) is the exponential of cross-entropy loss PPL = exp(loss), representing the effective branching factor per token. A PPL of 10 means the model is as uncertain as picking uniformly among 10 candidate words. Lower PPL indicates higher predictive confidence. Perplexity cannot be compared across different tokenizers without converting to Bits Per Byte."

Follow-ups to expect

• What is Bits Per Byte (BPB)? A tokenizer-agnostic metric that converts cross-entropy loss to bits per raw byte of text, allowing fair perplexity comparisons across models with different subword tokenizers.
• How is Perplexity used in prompt compression (LLMLingua)? LLMLingua measures per-token perplexity under a small model: tokens with very low perplexity (highly predictable filler words) are pruned, while high-perplexity tokens (informative keywords) are retained.