After pretraining, align the model with human preferences using a reward model + policy gradient (PPO) or DPO.

Data Flow

  prompts
    |
    |--> base model --> response A, response B
    |
    v
  human ranker labels (A better than B)
    |
    v
  reward model R(x, y) trained to predict human ranking
    |
    v
  base model + R signal --> PPO update --> aligned model

What it does

• Reduces harmful / unhelpful outputs.
• Improves instruction-following and tone.
• Does NOT add new facts — it reshapes distribution, not knowledge.

Pitfalls

• Reward hacking — the policy finds a quirk that scores high on R but is bad for users (e.g. always adds “Sure!”).
• PPO is unstable — hyper-parameter sweep matters.
• Modern alternative: DPO (Direct Preference Optimisation) avoids PPO entirely.

Analogy

A talented new hire who doesn’t yet know the house style. RLHF is the manager’s notes: “this reply feels cold; this one feels correct and warm”.

Interview tip: RLHF reshapes distribution, it does not add facts. Fine-tune for facts.