Skip to Content

BLEU and ROUGE (Text-Generation Metrics)

When to use

Score a text generation model (summariser, translator, paraphraser) against a reference answer.

Analogy

Comparing two essays by counting identical phrases. It does not check whether the arguments are the same, just whether the sentences look similar.

Data-flow diagram

   reference :  the cat sat on the mat
   generated :  the cat sat on a rug

   BLEU  = precision -- did gen's n-grams appear in ref?
   ROUGE = recall    -- did gen capture ref's n-grams?

Deep explanation

BLEU (Papineni 2002) is precision-based: n-gram overlap with a brevity penalty. ROUGE (Lin 2004) is recall-based — favoured for summarisation. Both measure lexical overlap, not semantic agreement. Pair with embedding cosine for higher-quality eval.

Examples

Example 1

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
ref = [['the','cat','sat','on','the','mat']]
hyp = ['the','cat','sat','on','a','rug']
print('BLEU:', sentence_bleu(ref, hyp,
        smoothing_function=SmoothingFunction().method1))

Smoothing avoids zero scores on short hypotheses.

Example 2

from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(['rouge1','rouge2','rougeL'], use_stemmer=True)
print(scorer.score('the cat is on the mat', 'the cat is on the mat'))

stemmer=True collapses word forms; rougeL finds the longest common subsequence.

Example 3

from sentence_transformers import SentenceTransformer
import torch
m = SentenceTransformer('all-MiniLM-L6-v2')
a, b = m.encode(['walked the dog', 'took the puppy for a stroll'], convert_to_tensor=True)
print('cosine:', torch.nn.functional.cosine_similarity(a, b, dim=0).item())

Sentence embeddings catch paraphrase-level agreement.

Common mistake

Trusting high BLEU/ROUGE on paraphrased output — these measure surface overlap, not meaning.

Key takeaway

BLEU and ROUGE are quick first-pass metrics for lexical fidelity. Pair with embedding cosine for semantic fidelity.

Production Failure Playbook

Failure scenario 1: high-bleu-factually-wrong

Failure scenario 2: tokenizer-mismatch-bleu