Convert raw text into integer IDs that the model’s embedding layer can index. Subword tokenisation strikes a balance between vocabulary size and sequence length.

Data Flow

text: "unhappiness"

BPE / WordPiece:
      un  happi  ness
      ^   ^      ^
     IDs 1023  876 5421

What it is

• BPE: starts with bytes, repeatedly merges most-frequent adjacent pair. GPT-2 / GPT-4 use this.
• WordPiece: merges chosen by likelihood gain, not raw count. BERT.
• SentencePiece: language-agnostic; trains from raw text including spaces. Llama.

Why subword?

• Word-level vocab: huge memory, no coverage for “unhappiness”.
• Char-level: very long sequences, weak signal.
• Subword: ~32k-100k tokens, near-perfect coverage, short sequences.

Code

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("gpt2")
ids = tok("unhappiness is rare").input_ids
# [10139, 318, 3127, 220, 816]
 
print(tok.decode(ids))
# 'unhappiness is rare'

Pitfalls

• Token IDs across models are incompatible — GPT-2 token 318 != Llama 3 token 318.
• Languages with non-Latin scripts tokenize into many more tokens than English — bill optimisations target this.

Analogy

Breaking a long word into common syllables that fit on a small set of wooden blocks.

Interview tip: Always state: token ID 318 means different things in different models. Candidates who miss this lose credibility fast.