Skip to Content

Perplexity (LM Surprisal)

When to use

Score how surprised a language model is by text — the standard intrinsic LM quality metric.

Analogy

Student preparing for a test. Low perplexity: I knew this material. High perplexity: I was guessing wildly.

Data-flow diagram

   text:  the cat sat on the
   PPL = exp( mean cross-entropy loss )
   lower = better (same tokenizer only)

Deep explanation

Perplexity is exp of cross-entropy loss: the per-token ‘how surprised was I’ averaged over the sequence. Compare perplexity only between models that share the same tokenizer and vocabulary. Modern generative LLMs are often evaluated by downstream benchmarks instead.

Examples

Example 1

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer
tok = GPT2Tokenizer.from_pretrained('gpt2')
mdl = GPT2LMHeadModel.from_pretrained('gpt2')
ids = tok.encode('the cat sat on the mat', return_tensors='pt')
with torch.no_grad():
    loss = mdl(ids, labels=ids).loss
print('perplexity =', torch.exp(loss).item())

HuggingFace loss is mean cross-entropy per token; exp(loss) gives perplexity.

Example 2

import numpy as np
p = np.array([0.1, 0.4, 0.3, 0.2])
q = np.array([0.2, 0.3, 0.3, 0.2])
ce = -np.sum(p * np.log(q + 1e-12))
print('perplexity =', np.exp(ce))

NumPy version shows PPL is just exp(mean NLL) — useful when no model is loaded.

Example 3

@torch.no_grad()
def eval_ppl(model, loader):
    losses = []
    for x, y in loader:
        with torch.amp.autocast(device_type='cuda', dtype=torch.bfloat16):
            loss = model(x.cuda(), y.cuda()).loss
        losses.append(loss.item())
    return float(torch.exp(torch.tensor(losses).mean()))

Karpathy-style eval loop aggregates over batches — canonical in training scripts.

Common mistake

Comparing perplexity across models with different tokenizers. Vocab size and token granularity change the metric.

Key takeaway

Perplexity equals exp of cross-entropy. Lower is better, identical tokenizers only. Pair with downstream benchmarks for task-level quality.

Production Failure Playbook

Failure scenario 1: garbage-in-perplexity

Failure scenario 2: ppl-drops-toxicity-spikes