How to fine-tune a model with Direct Preference Optimization (DPO)

DPO trains models to prefer good responses over bad ones using human preference data — without the complexity of reinforcement learning. This guide covers collecting preference data, training with TRL's DPO trainer, and evaluating results.

SFT vs DPO: when to use which

Supervised Fine-TuningSupervised Fine-TuningSupervised Fine-Tuning (SFT) is the process of training a pre-trained base model on curated instruction-response pairs to teach it specific tasks before preference alignment.Learn more → (SFT) teaches the model to produce specific outputs for specific inputs. Direct Preference OptimizationDirect Preference OptimizationA training method that aligns language models with human preferences directly from preference pairs, bypassing the need for separate reward models or reinforcement learning.Learn more → (DPO) teaches the model to prefer certain types of outputs over others — making it better at following instructions, being helpful, or matching your brand voice — without specifying exact outputs.

Use SFT when you have a clear correct answer (classification, extraction, specific formats). Use DPO when you want to improve the model's judgment and style — reducing verbosity, improving tone, better handling of refusals, or aligning with your organisation's communication style.

Collect preference data

DPO requires paired preference data: for each promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →, a 'chosen' response (the better one) and a 'rejected' response (the worse one). The format is `{"prompt": "...", "chosen": "...", "rejected": "..."}`. Aim for 1000+ pairs for reliable results.

The most efficient way to collect pairs: run your base model on 500 prompts, generating 2 different responses per prompt (use `n=2` in the API call), then have human raters choose the better response. Alternatively, use a stronger model as an automatic judge to rank pairs from your weaker model.

Set up the DPO trainer

Install the required packages: `pip install trl transformers peftPEFTParameter-Efficient Fine-Tuning (PEFT) adapts large language models by training only a small subset of parameters, dramatically reducing computational costs while maintaining performance.Learn more → accelerate bitsandbytes`. Load your base model with 4-bit quantisation: `from transformers import AutoModelForCausalLM; model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-Instruct-v0.3', load_in_4bit=True)`.

Configure the DPO trainer: `from trl import DPOTrainer, DPOConfig; config = DPOConfig(beta=0.1, max_length=2048, output_dir='dpo-output'); trainer = DPOTrainer(model=model, args=config, train_dataset=dataset, tokenizer=tokenizer)`. The `beta` parameter controls how strongly the model is pushed away from the rejected responses — 0.1 is a good starting value.

Train and evaluate

Run training: `trainer.train()`. DPO training is generally faster than SFT because datasets are smaller. On a single A100 with a 7B model, 1000 pairs takes around 30–60 minutes at 4-bit quantisation.

Evaluate by running your test prompts through both the base model and the DPO-trained model, then using an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → judge to compare outputs. Good DPO training should show: more concise responses if you trained for that, stronger instruction following, and reduced occurrence of the specific failure patterns captured in your 'rejected' examples.

Common pitfalls

Low-quality preference labels are the biggest failure mode. If raters were inconsistent about what 'better' means, the model learns noise. Provide clear annotation guidelines before collecting data: specify what makes a response better in measurable terms.

Catastrophic forgettingCatastrophic ForgettingThe phenomenon where neural networks lose previously learned knowledge when trained on new tasks or data, requiring careful strategies to preserve existing capabilities.Learn more → is worse with DPO than SFT. Always evaluate general capabilities after training. If the model regresses on tasks unrelated to the preference data, reduce `beta` or add a small SFT mix to the training data to anchor general capabilities.