RLHF.
Reinforcement Learning from Human Feedback — a training technique that uses human preferences to teach AI models to be helpful, honest, and harmless.
RLHF is the training technique that transformed raw language models into the helpful assistants we use today. The process begins by collecting human feedback: annotators compare pairs of model outputs and indicate which one is better. A reward model is trained to predict human preferences, and then the LLM is fine-tuned using reinforcement learning to maximize the reward model's score.
RLHF was key to making ChatGPT dramatically more useful than earlier GPT models. The same underlying model weights, shaped by human feedback, became far more helpful, less likely to produce harmful content, and better at following instructions. OpenAI, Anthropic, and DeepMind all use RLHF (or variants like RLAIF, Constitutional AI, and DPO) as core to their alignment techniques.
RLHF has known failure modes: reward hacking (the model learns to game the reward model), sycophancy (preferring flattering but incorrect answers), and alignment to annotator biases rather than universal human values. These challenges motivate ongoing research into improved alignment techniques.
Related terms.
- RLHFReinforcement Learning from Human Feedback — a training technique that uses human preferences to teach AI models to be helpful, honest, and harmless.
- Instruct ModelA base language model that has been fine-tuned to follow instructions from users, as opposed to just completing text in the style of its training data.