Summarization & ROUGE/BLEU

Evaluating Text Generation

How do you grade a computer generated summary?

Unlike classification tasks where answers are right or wrong, a single document can have dozens of equally valid human summaries.

Automated evaluation relies on two primary traditional metric families:

┌──────────────────────────┬──────────────────────────┐
│ 1. BLEU (Precision-Based)│ 2. ROUGE (Recall-Based)  │
├──────────────────────────┼──────────────────────────┤
│ Focuses on PRECISION.    │ Focuses on RECALL.       │
│ "Are the words generated │ "Did the model capture   │
│ by the model correct?"   │ all key reference facts?"│
│ Standard for Translation │ Standard for Summarizing │
└──────────────────────────┴──────────────────────────┘

1. BLEU (Bilingual Evaluation Understudy)

BLEU calculates modified n-gram Precision between candidate text $C$ and reference text $R$:

$$\text{BLEU} = \text{BP} \cdot \exp\left( \sum_{n=1}^N w_n \log p_n \right)$$

• $p_n$: Precision of $n$-grams (unigrams, bigrams, trigrams, 4-grams).
• Brevity Penalty (BP): Penalizes models that generate short 2-word sentences to cheat precision scores.

$$\text{BP} = \begin{cases} 1 & \text{if } c > r \ \exp\left(1 - \frac{r}{c}\right) & \text{if } c \le r \end{cases}$$

2. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

ROUGE focuses on Recall: out of all $n$-grams present in the human reference summary, how many did the candidate summary catch?

Common ROUGE Variants

1. ROUGE-1: Unigram (single word) overlap recall. Measures basic topic coverage.
2. ROUGE-2: Bigram (two word pair) overlap recall. Measures phrase fluency and factual structure.
3. ROUGE-L: Longest Common Subsequence (LCS). Measures in-order structural similarity without requiring consecutive matches.

  Reference: "The quick brown fox jumps over the dog"
  Candidate: "A quick brown fox jumped over a dog"

  ROUGE-1 Recall: 6 matching words / 8 reference words = 0.75
  ROUGE-2 Recall: 3 matching bigrams ("quick brown", "brown fox", "over the") = 0.43

Modern Neural Evaluation (Beyond Surface Overlap)

N-gram metrics suffer from Synonym Blindness: if a human writes "automobile" and an LLM writes "car", ROUGE scores it as a complete mismatch ($0.0$).

Modern pipelines use:

1. BERTScore: Computes pairwise cosine similarity between BERT contextual embeddings of candidate and reference words.
2. G-Eval (LLM-as-a-Judge): Prompts GPT-4 with explicit rubrics to grade summaries on Fluency, Relevance, Consistency, and Coherence.

Say this out loud

BLEU measures n-gram precision and is standard for translation. ROUGE measures n-gram recall and is standard for text summarization to verify that key facts were retained. ROUGE-1 evaluates single words, ROUGE-2 evaluates bigrams, and ROUGE-L evaluates Longest Common Subsequence. Modern evaluation supplements n-gram metrics with BERTScore and LLM judge rubrics.

Followups to expect

1. What is Extractive vs Abstractive Summarization? Extractive summarization copies key sentences directly from the source text. Abstractive summarization rephrases concepts in new words using generative LLMs.
2. What is Faithfulness / Hallucination in Summarization? Measuring whether generated summary facts are strictly supported by the source document without inventing un-grounded details.