Skip to Content

Building LLMs — Lecture deep-dive

When to use

You are about to design, evaluate, or steer a Large Language Model product and keep hitting buzzwords like perplexity, BPE tokenization, RLHF, DPO, Chinchilla, HNSW. Skim this page when you need a one-stop, beginner-friendly walkthrough of how LLMs are actually built and evaluated.

Analogy

Think of an LLM as a language factory. The factory has four rooms:

Each room has its own tools, costs, and failure modes. Forget one and the whole factory grinds to a halt.

Data-flow diagram

       internet (250 B pages ~ 1 PB)
                  |
   crawl -> extract text -> filter PII / dedupe / heuristic quality
                  |
       Common Crawl ~ 100 B cleaned tokens
                  |
              pretrain  (transformer, next-token cross-entropy)
                  |
        base model (e.g. Llama-3 405 B, 15.6 T tokens)
                  |
     supervised fine tuning (SFT, ~10-50 K examples)
                  |
         assistant model that follows instructions
                  |
       RLHF / DPO  (preferences over pairs, ~1 M labels)
                  |
            shipped model  (e.g. ChatGPT)

Deep explanation

A modern LLM is built in two phases: pretraining (a self-supervised exercise on a huge corpus) and post-training (supervised + preference fine-tuning that turns a language modeler into an assistant). Pretraining costs millions of dollars and trains on trillions of tokens; post-training costs a fraction but changes how the model behaves.

The whole pipeline:

  1. Tokeniser (BPE) cuts text into sub-word units (≈ 3–4 letters each) so the model sees a fixed vocabulary instead of a never-ending list of typos.
  2. Pretraining runs cross-entropy loss: given tokens 1..t, predict token t+1. The architecture is one transformer; the magic is scale (parameters × data × compute).
  3. Evaluation uses perplexity in development and aggregate benchmark suites (MMLU, HELM, ChatBotArena) for academic comparison.
  4. SFT trains the model on a small set of “good answers” written by humans (or by another LLM).
  5. RLHF/DPO trains the model on human preference pairs — “is A better than B for this prompt?” — so the model learns what users actually want, not just what is grammatically likely.
  6. Systems carries the whole thing on GPU clusters: low precision, operator fusion, sharding — all to keep Model FLOPs Utilization (MFU) as high as possible.

Pretraining — what are we actually fitting?

Definitions in one line each

Simple worked example

The sentence “the mouse ate the cheese” becomes five token IDs [t_1, t_2, t_3, t_4, t_5]. The model predicts a probability distribution over the 100 K vocabulary for what follows t_1..t_4; if the real answer is t_5, the cross-entropy contribution is $-\log p(t_5)$. Average over millions of sentences and minimise the total — that is what training does.

Why we use cross-entropy

Cross-entropy between the predicted distribution and the one-hot “real answer” distribution is mathematically identical to maximising the log-likelihood of the corpus. One function fits both views.

The bitter lesson, in one sentence

Architectures matter less than you think; scale + data + systems matter more. Always-on scaling laws predict loss from compute / data / params, so we tune on small models before training a big one.

Tokenisation — Byte-Pair Encoding (BPE)

One-line summary

Start with characters; repeatedly merge the most frequent adjacent pair into a new token. After ~100k merges you have a vocabulary of frequent chunks, slightly bigger than a character.

Worked example

Start: each character is its own token. After merges:

A real GPT-3 tokenizer splits tokenizer into [token, izer]. The vocabulary size is the size of the output dimensionality, so changing the tokenizer mid-project is very painful.

Three gotchas

  1. Typos: tokenisers pre-merge so a typo doesn’t become <unk>. They still degrade quality.
  2. Numbers: 327 may be a single token, so the model is bad at multi-digit arithmetic. Companies now treat numbers specially.
  3. Code whitespace: 4-space indents deserve their own token; GPT-4 fixed this and code generation improved.

Evaluation

Perplexity

Aggregated benchmarks

Two specific evaluation gotchas

  1. Methodology drift: Llama-65B scored 63.7 on HELM and 48.8 on a different benchmark for the same task. Pick one benchmark and stick with it.
  2. Train/test contamination: the test set might already be in pre-training data. People detect this by comparing the model’s likelihood of the test items in the canonical order vs shuffled (it shouldn’t matter unless the model memorised them).

Data

The recipe at a glance

  1. Crawl the open web (Common Crawl gives ~250 B pages / 1 PB).
  2. Extract text from HTML (very fiddly for math, code, tables).
  3. Filter:
  4. De-duplicate exact and fuzzy (MinHash) — 30-50% of crawl is repeated headers/footers.
  5. Domain-mix: up-weight code, books, math, QA; down-weight entertainment.
  6. High-quality tail: at the end of training, anneal the learning rate and over-fit on Wikipedia + curated data.

Why data is the secret

The same architecture trained on the same compute budget with better data beats a bigger architecture with worse data. In the Llama-2/3 team of ~70 researchers, ~15 worked on data. The rest did modelling + systems.

Scaling Laws

Three lawyers: compute, data, parameters

If you plot loss vs log(compute) / log(data) / log(params) you get a straight line. That means you can extrapolate from a small model how loss will scale at 10× or 100× the compute.

Chinchilla / Inference-aware

For a fixed compute budget, the optimal models follow:

Why you bother

Old workflow: train 30 models for 1 day each, pick the best. New workflow: train many small models (1 day × ~3 sizes), fit a scaling law, extrapolate, then train one final huge model for ~27 days. Predictability replaces brute force.

Back-of-the-envelope cost of a frontier model

For Llama-3-405B trained on 15.6 T tokens:

Post-training: from “internet autocomplete” to assistant

SFT — Supervised Fine-Tuning

Why SFT alone isn’t enough

RLHF (Reward model + PPO)

  1. For each prompt, generate two answers (A, B).
  2. Human raters pick the better.
  3. Train a reward model to predict the human’s choice (Bradley-Terry logits).
  4. PPO adjusts the LM to maximise the reward model — with a KL penalty against the original model to avoid “reward hack” mode collapse.

DPO — Direct Preference Optimisation

Why RLHF can produce rambling answers

Human raters prefer longer text. The model learns: “be longer”. This is a measurable bias in ChatBotArena logs and is fixed in modern eval suites with length regression.

Systems — keeping the factory alive

The constraints

Two tricks that buy you 2-4×

  1. Low precision: store weights in 32-bit floats, compute in 16-bit (or 8-bit with care). Memory traffic halves; throughput doubles. Training uses automatic mixed precision: weights saved at 32-bit, activations at 16-bit.
  2. Operator fusion: every x = cos(x); x = exp(x); x = sin(x) round-trips to global memory. torch.compile rewrites PyTorch into one fused CUDA kernel; one round-trip, 2× faster.

MFU (Model FLOPs Utilization) and why labs obsess over it

The exact formula:

MFU = observed_FLOPs_per_sec / theoretical_peak_FLOPs_per_sec

In words: divide the FLOPs you actually ran per second by the FLOPs the silicon could run in a perfect world. Range = [0, 1]. Real labs run at 0.40–0.50; 0.50 is excellent; > 0.60 is world-class, usually only inside a single tightly-tuned kernel. The gap to 1.0 mostly comes from communication (data hops between cores / nodes). If your code does 1 matmul then 1 softmax then 1 sigmoid, each line waits for an HBM round-trip — that gap is exactly what MFU measures.

3D parallelism at a glance (DP × TP × PP)

Most frontier runs combine three parallelisms on the same cluster. Picture 8 nodes × 8 GPUs = 64 GPUs in total:

            DP=4 (replicate model 4× across nodes, split data)
   ┌─────────────────────────────────────────────────────────────┐
   │   ├── DP-rank 0 ──┐                                          │
   │   ├── DP-rank 1 ──┤   TP=2  (split each weight matrix       │
   │   ├── DP-rank 2 ──┤    across 2 GPUs within one node)        │
   │   └── DP-rank 3 ──┘                                          │
   │                                                             │
   │  PP=4 stages along the depth direction:                       │
   │   stage 0  ▓▓▓  ..  ..  ▓▓▓      bubble at warm-up / drain   │
   │   stage 1   ..  ▓▓▓  ..  ▓▓      bubble same problem          │
   │   stage 2   ..  ..  ▓▓▓  ▓▓      bubble same problem          │
   │   stage 3   ▓▓  ..  ..  ▓▓▓      bubble same problem          │
   │   (▓ = real work, .. = idle GPU — that's the bubble)         │
   └─────────────────────────────────────────────────────────────┘

The bubble is the fraction of the timeline when a pipeline-parallel stage is waiting on its neighbour. Shrink the bubble by either (a) more micro-batches in flight, or (b) fewer PP stages (which pushes more work onto tensor-parallel, which costs more all-reduce). Every frontier lab tunes this DP × TP × PP grid per workload — there is no single best answer.

Tiling: keep matrices resident on a single core’s high-bandwidth SRAM

Big matrices don’t fit in on-chip SRAM (a few tens of MB). Cube cores load a TILE — say 128 × 128 — into shared SRAM, do all the work locally, write back. Tiling is how 24× faster is reached on memory-bound kernels without changing the algorithm.

Parallelism: three flavours, three trade-offs

When you pick which

Production failure playbook

Failure scenario 1: a great model ships and forgets everything about itself

Failure scenario 2a: offline scaling plateau

Failure scenario 2b: online A/B rubric drift

Failure scenario 3: a “perfect” benchmark score is a train/test leak

Common mistake

Key takeaway

A frontier LLM is a 4-stage factory: collect clean data, pretrain a transformer, fine-tune it to follow instructions, prefer the answers humans actually want. Each stage has its own failure mode (data drift, eval contamination, KL collapse, RL alignment). Knowing the factory by heart beats knowing the arXiv paper of the week.

Topics in this lecture

  1. When to use (where this page fits)
  2. Autoregressive language modelling
  3. Tokenisation (BPE)
  4. Perplexity + MMLU
  5. Data recipe
  6. Chinchilla + scaling laws
  7. SFT
  8. RLHF + DPO
  9. Systems (precision + fusion)
  10. Production failure playbook