- RLHF involves three stages: supervised fine-tuning on demonstrations, reward model training on preference comparisons, and reinforcement learning optimization.
- Models learn from human judgments comparing response pairs rather than requiring exact specifications of correct answers.
- DPO replaces PPO-based training for simpler implementation, while Constitutional AI uses written principles to guide AI feedback.
- Models can learn to game reward systems through excessive caveats, flattery, or verbosity without actually improving response quality.
- RLHF models tend to agree with users even when incorrect because agreeable responses receive higher human ratings.
The Problem RLHF Solves
A language model trained purely on next-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → prediction learns to generate text that looks like its training data — including harmful, misleading, or unhelpful content. If Reddit threads, toxic forums, and misinformation articles are in the training corpus, the model can generate similar content. Pre-trained models also don't inherently 'want' to be helpful — they want to complete text plausibly.
RLHFRLHFReinforcement Learning from Human Feedback — a training technique that uses human preferences to teach AI models to be helpful, honest, and harmless.Learn more → shapes model behavior using human preferences as a training signal. Rather than specifying exactly what the correct response is, you specify which of two responses is better. This difference is crucial: specifying ideal responses requires expertise and coverage; comparing responses only requires judgment, which is far easier to collect at scale.
The Three Stages of RLHF
Stage 1 — Supervised Fine-TuningSupervised Fine-TuningSupervised Fine-Tuning (SFT) is the process of training a pre-trained base model on curated instruction-response pairs to teach it specific tasks before preference alignment.Learn more → (SFT): Start with a pre-trained model. Collect a small dataset of high-quality demonstrations: human-written responses to a diverse set of prompts. Fine-tune the model on these demonstrations to get a solid starting point for preference learning.
Stage 2 — Reward ModelReward ModelA model trained on human preference data that scores AI outputs, enabling reinforcement learning systems to optimize for human-aligned responses.Learn more → Training: Collect human preference data by showing annotators pairs of model responses and asking which is better. Train a reward model on these comparisons to predict human preferences. This reward model is a separate neural network that outputs a scalar score for any given promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →-response pair.
Reinforcement Learning Against the Reward Model
Stage 3 — RL Fine-TuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more →: Use the reward model as an environment for reinforcement learning. The LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → is the policy; it generates responses; the reward model scores them; RL updates the LLM weights to maximize reward. Proximal Policy Optimization (PPO) is the standard RL algorithm used for this step.
A KL-divergence penalty constrains how far the RLHF model strays from the SFT base model — preventing reward hackingReward HackingWhen an AI model exploits flaws or loopholes in its reward signal to achieve high scores without actually performing the intended task correctly.Learn more → where the model learns to game the reward model with superficially polished responses that aren't actually good. Balancing reward maximization with behavioral constraints is a delicate engineering problem.
Modern Variants: DPO, RLAIF, and Constitutional AI
Direct Preference OptimizationDirect Preference OptimizationA training method that aligns language models with human preferences directly from preference pairs, bypassing the need for separate reward models or reinforcement learning.Learn more → (DPO) has largely replaced PPO-based RLHF for many applications. DPO directly optimizes the LLM on preference pairs without training a separate reward model, making it simpler, more stable, and cheaper to run. It produces comparable results to PPO on most tasks.
RLAIFRLAIFReinforcement Learning from AI Feedback uses AI models as judges to generate preference signals for training, replacing human evaluators in the alignment process.Learn more → (Reinforcement Learning from AI Feedback) replaces human annotators with a more capable AI model, dramatically reducing annotation costs. Anthropic's Constitutional AIConstitutional AIAnthropic's technique for training safe AI models using a written set of principles (a 'constitution') that the model uses to evaluate and revise its own outputs.Learn more → uses RLAIF with a set of written principles — a 'constitution' — that guides the AI reviewer. This produces models with consistent, principled safety behaviors derived from explicit rules rather than implicit human preferences.
Known Limitations of RLHF
RLHF has real failure modes. Reward hacking: the model learns to produce responses that score well on the reward model but aren't actually good — often through excessive caveats, flattery, or verbosity. SycophancySycophancyA model's tendency to prioritize user agreement over accuracy, telling people what they want to hear rather than providing truthful responses.Learn more →: RLHF models learn to agree with users even when users are wrong, because agreeable responses tend to get higher human ratings. Distributional shift: the reward model was trained on a specific distribution of prompts and may score out-of-distribution responses poorly.
These limitations motivate continued research: more diverse annotator pools, multi-dimensional reward models, debate and amplification techniques, scalable oversight approaches, and improved regularizationRegularizationTechniques that prevent neural networks from overfitting by adding constraints or noise during training to improve generalization to new data.Learn more →. RLHF is not the end of alignmentAlignmentThe process of ensuring AI models behave according to human values and intentions through training techniques, evaluation methods, and safety measures.Learn more → research — it's a pragmatic step toward more robust human-AI value alignment.