Skip to content
LearnMLByDoing
Free

4.6 Alignment: reward models, RLHF and DPO

36 minProadvancedInstructGPTDPOCompendiumPPO

Preference tuning pushes a fine-tuned model toward answers people prefer while a KL leash keeps it near where it started, and the leash's strength decides whether it learns to help or learns to game the reward model.

01Pretraining, SFT, and what a policy is

A chat model is built in three stages. Pretraining teaches a base model to continue web text by next-token prediction (lesson 4.4). Supervised fine-tuning (SFT) trains it further on tens of thousands of written demonstrations, prompts paired with good answers, so it answers instead of continuing. Preference tuning, the subject of this lesson, then teaches it which of its own answers people prefer.

In reinforcement-learning language the model is a policy: given a prompt xx it puts a probability on every complete response yy. That probability is the product of its next-token probabilities, so its log is a sum you can compute in one forward pass:

log⁡πθ(y∣x)=∑t=1Tlog⁡pθ(yt∣x, y<t)\log \tok{pi_theta}{\pi_\theta}(y\mid x)=\sum_{t=1}^{T}\log p_\theta\big(y_t\mid x,\,y_{<t}\big)
A response's log-probability: the sum of its tokens' log-probabilities.
log π_θ(y | x) for a batch[,]

Gather the log-probability of each actual next token from the logits, mask the prompt and padding, then sum over T to get one number per response, shape [B]. RLHF and DPO both work with these sums.

The viewport shrinks this to one prompt and 625 candidate responses, laid out on a floor by two features: x1x_1, how much substance a response has, and x2x_2, how long and flattering it is. Heights are π\sqrt{\pi}, so a policy that piles all its mass on one response still fits on screen. The emerald surface starts as the broad pretrained model and SFT pulls it onto the cyan wire, πref\tok{pi_ref}{\pi_{\mathrm{ref}}}, while cyan demonstrations land on the floor. The HUD's KL(πθ∥πref)\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}}) falls from 0.41 nats to 0.00 by step 120.

SFT can only copy what demonstrators wrote. It cannot express "this answer is better than that one", and writing ideal answers is slower than comparing two. The SFT model is where preference tuning starts, and it stays in the picture as πref\pi_{\mathrm{ref}}, the anchor that every later step is measured against.

Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.1 and 3.5

Maths, CS & AI Compendium, Henry Ndubuaku, LLM post-training

02Preference data: which answer is better?

A preference dataset is built by sampling two or more responses to the same prompt, usually from the SFT model, and asking a rater which is better. One record looks like this:

prompt Can I take ibuprofen with my blood-pressure medication? chosen Ibuprofen can raise blood pressure and weaken some blood-pressure medicines, so ask your pharmacist or doctor before combining them. rejected Yes, ibuprofen is safe to take with any medication.

preference dataset[,]

InstructGPT's reward model was trained on roughly 33k prompts, each with 4 to 9 ranked responses; a ranking of K responses yields K(K−1)/2 pairs.

Comparisons are used instead of scores out of ten because people disagree about absolute scales but agree more often about which of two answers is better. They still disagree a lot: InstructGPT reports that its labelers agreed with each other about 73% of the time. Labels are noisy, and the model of them has to allow for that.

The viewport is running RLHF, which starts with this data. Its 400 pairs are drawn from πref\pi_{\mathrm{ref}}, so they cluster under the cyan wire; each is a floor segment with an emerald chosen end and a grey rejected end, and 24 of them are on screen at a time. Their labels come from the raters' preferences in the toy: the share of raters choosing aa over bb is σ(r^a−r^b)\sigma(\hat r_a-\hat r_b), so pairs near the reference are close calls, between 50% and 73% agreement.

Restart the RLHF run and hold it at step 0 to read the data before any training.

Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.3 and 3.4

03Reward models and the Bradley–Terry loss

A reward model rϕ(x,y)r_\phi(x,y) is usually the SFT network with its vocabulary head replaced by a single scalar output. It is trained so that the difference of two scores predicts the preference, through the Bradley–Terry model:

P(yw≻yl∣x)=σ(r(x,yw)−r(x,yl))=er(x,yw)er(x,yw)+er(x,yl)P(\tok{pref}{y_w\succ y_l}\mid x)=\sigma\big(\tok{reward}{r}(x,y_w)-\tok{reward}{r}(x,y_l)\big)=\frac{e^{r(x,y_w)}}{e^{r(x,y_w)}+e^{r(x,y_l)}}
σ is the logistic sigmoid of lesson 0.7. Only the difference of the scores matters.
The loss and its gradient1/4
  1. 1
    L(ϕ)=−E(x,yw,yl)[log⁡σ(Δ)],Δ=rϕ(x,yw)−rϕ(x,yl)\mathcal{L}(\phi)=-\mathbb{E}_{(x,y_w,y_l)}\big[\log\sigma(\Delta)\big],\qquad \Delta=r_\phi(x,y_w)-r_\phi(x,y_l)

    Maximum likelihood of the observed choices: the negative log-probability the model gave to what the rater chose.

Loading figure
Drag the scores A and B, or press Gradient step. The amber arrows are the descent direction −∂L/∂r-\partial\mathcal{L}/\partial r: both scores move by the same amount in opposite directions, and the steps shrink as PP approaches 1. Shift both scores and nothing else changes. Pair 3 is the habit from the last step: both answers are right and the rater chose the longer one.

In the viewport, RLHF stage 1 of 2 fits the reward model for 60 steps while the policy waits. The toy reward model is a weighted sum of two features of a response: a broad helpful hill at (0.6, 0.3)(0.6,\ 0.3) and a narrow bump at (1.5, 1.6)(1.5,\ 1.6), long and flattering answers. The fit recovers weights (0.998, 1.397)(0.998,\ 1.397) for the raters' (1.0, 1.4)(1.0,\ 1.4), and the amber surface r^\tok{reward}{\hat r} grows from flat into two hills; the bump is the artefact peak. The loss only falls from 0.693 to 0.681, because pairs drawn near πref\pi_{\mathrm{ref}} are close calls and their labels are close to 50/50.

Restart RLHF with β=0.1\beta = 0.1 and watch the first two seconds.

Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022, Sec. 3.5, Reward modeling

Maths, CS & AI Compendium, Henry Ndubuaku, Logistic regression and the sigmoid

The rest of this lesson is in Pro

  • 04RLHF: maximise reward on a KL leash
  • 05The closed-form optimum, and DPO
  • 06Reward hacking, over-optimisation and the role of β
  • 07Variants and practice
  • Python challenge: Bradley–Terry, DPO and the KL-optimal policy
  • Knowledge check, 4 questions

Pro opens all 45 lessons, every graded Python challenge and the full 3D labs.

Compiling scene

The Python challenge and knowledge check for this lesson are part of Pro.

Questions learners ask

What is the difference between RLHF and DPO?
Both optimise the same objective: expected reward minus a KL penalty that keeps the model close to its supervised starting point. RLHF first trains a separate reward model on human preference pairs and then uses reinforcement learning, usually PPO, to sample responses and push the model toward high scores. DPO uses the closed-form solution of that objective to rewrite the reward in terms of the model's own probabilities, so it trains directly on the preference pairs with a supervised-style loss, with no reward model, no sampling and no value network. RLHF learns from the model's own new outputs; DPO is simpler and cheaper but learns only from the fixed dataset.
What does the KL penalty (beta) do in RLHF?
It charges the model for drifting away from the reference model it started from. The reward model is only reliable on the kind of answers it was trained on, and the reference model writes those answers, so the penalty keeps the policy where the reward means something and preserves its fluency and knowledge. A large beta keeps the model almost unchanged; a small beta lets it chase the reward model's highest scores, including its mistakes. The optimal policy for a given beta is the reference model reweighted by exp(reward / beta).
What is reward hacking in language models?
Reward hacking, or reward over-optimisation, happens when a model finds responses that a learned reward model scores highly but people do not actually prefer, such as overly long, flattering or evasive answers. The reward-model score keeps rising while real quality falls, which is Goodhart's law. It is controlled with a stronger KL penalty or early stopping, better and fresher preference data, reward-model ensembles, length control, and evaluation by humans or independent judges on held-out prompts.

Next in Transformers and LLMs

4.7LLM agents, tool use and evaluationReason, act, observe: the agent loop, function calling, planning budgets, and how to evaluate a system that takes actions.

Sources

  • Training Language Models to Follow Instructions with Human Feedback, Ouyang et al., 2022
  • Direct Preference Optimization, Rafailov et al., 2023
  • Maths, CS & AI Compendium, Henry Ndubuaku
  • Proximal Policy Optimization Algorithms, Schulman et al., 2017