Quality beats quantity
A fine-tuned model can only be as good as its training data. 200 high-quality, diverse, correctly formatted examples will outperform 2000 noisy or redundant ones. Before collecting data, define what a 'good' example looks like and create evaluation criteria you can apply consistently.
Common mistakes: including outputs that are factually wrong, including duplicate or near-duplicate examples, mixing multiple task types without labelling them, and using examples that are too short to teach the model meaningful patterns.
Data collection strategies
Synthetic dataSynthetic DataModel- or program-generated training data used to augment or replace human-collected data, central to modern AI training and post-training processes.Learn more → from a stronger model: use GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o or Claude Opus to generate instruction-response pairs for your task. Write 20–30 diverse seed prompts manually, then use the model to generate variations. Review and filter the outputs — expect to keep 60–80% of generated examples.
Human annotation: the gold standard. Have domain experts write or rate responses. Use annotation guidelines that specify what makes a good response for your task. Aim for 2+ annotators per example and measure inter-annotator agreement.
DistillationDistillationA technique where a smaller student model is trained to replicate the behavior of a larger teacher model, achieving similar performance with reduced computational requirements.Learn more → from existing data: if you have logs of user queries and good responses (e.g. from a GPT-4 deployment), filter them for quality and use them directly. This is often the fastest path to a good dataset.
Formatting standards
Use the model's native chat template. For Llama-3-based models, format examples as `<|begin_of_text|><|start_header_id|>system<|end_header_id|>...` — use the tokenizer's `apply_chat_template` method rather than manual string formatting to get it exactly right.
The Alpaca format (instruction / input / output) is a widely supported alternative for simpler tasks: no system message, just the instruction and the model's response. Many fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → tools and datasets use this format.
Quality filtering
Filter out examples shorter than 50 tokens (too little signal) or longer than your max sequence length (will be truncated). Remove examples where the response is a refusal ('I can't help with that') unless you are specifically teaching refusal behaviour.
Use perplexityPerplexityA statistical measure of how well a language model predicts text — lower perplexity means the model finds the text less surprising and is a better predictor.Learn more → filtering: run your base model on each example and compute the loss. Very high perplexity examples are either very hard or wrong; very low perplexity examples are too easy and add little value. Remove the extremes and keep the middle.
Minimum viable dataset sizes
Style and tone adaptation: 100–200 examples. The model is already capable of the task; you are just adjusting the output style.
New task or domain: 500–1000 examples minimum. Below this, the model may not generalise reliably.
Complex multi-step tasks: 2000+ examples with good coverage of edge cases. Include examples of failure modes and correct handling.
Always hold out 10% of your data as a validation set. Monitor validation loss during training — if it starts increasing, you are overfittingOverfittingWhen a machine learning model memorizes training data too closely, performing well on training examples but failing to generalize to new, unseen data.Learn more →.