- Continues training on curated datasets after pre-training to specialize models for specific tasks or behaviors.
- Adds small adapter matrices to transformer layers, requiring 1/100th the GPU memory of full fine-tuning.
- SFT trains on prompt-response pairs while RLHF uses preference comparisons to shape behavior.
- Use when you need consistent format, have hundreds of examples, or require domain knowledge injection.
- LLM-as-judge evaluations using GPT-4 or Claude have become the standard for assessing generation quality.
- Fine-tuning can degrade general capabilities, requiring regular monitoring of base model performance.
What Fine-Tuning Actually Does
Fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → continues the training process on a smaller, curated dataset after a model has been pre-trained on web-scale data. Rather than starting from random weights, fine-tuning starts from the pre-trained weights and adjusts them slightly to specialize the model for a target task, domain, or behavior. The result inherits the general capabilities of the base model while excelling in the specialized area.
Fine-tuning can teach a model to always respond in a specific format, adopt a particular writing style, answer questions about a proprietary domain, or refuse certain types of requests. It's most useful when prompt engineeringPrompt EngineeringThe practice of crafting inputs, instructions, examples, and formatting to guide language models toward producing better, more accurate outputs.Learn more → alone can't achieve consistent behavior at the level of quality you need.
When to Fine-Tune (and When Not To)
Fine-tune when: you need consistent output format or style that's difficult to enforce through prompting, you have hundreds of task-specific examples, you need to inject domain knowledge not in the base model, latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → requirements are tight (fine-tuned smaller models can match prompted larger ones), or you need to embed behaviors that system prompts can't reliably produce.
Don't fine-tune when: you're still figuring out what the model should do (iteration via prompting is faster), you have fewer than ~50–100 examples (the model will overfit), the task is one the base model already does well, or you can achieve your goal with RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → or good prompting. Many teams find that 90% of their use cases don't require fine-tuning.
LoRA: The Standard Fine-Tuning Method
Low-Rank Adaptation (LoRALoRALow-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains small weight matrices while freezing the base model.Learn more →) is the dominant technique for efficient fine-tuning. Instead of updating all model weights (which requires as much memory as training from scratch), LoRA adds small adapter matrices to each transformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more → layer. These adapters have a low-rank structure (rank 4–16 is typical), meaning they have far fewer parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → than the full weight matrices they augment.
In practice, LoRA fine-tuning requires 1/100th the GPU memory of full fine-tuning. A Llama 3.3 70B LoRA can be trained on 4× A100 GPUs in hours rather than requiring a cluster for days. QLoRA (quantized LoRA) reduces requirements further — fine-tuning a 70B model on a single 48GB GPU is feasible with 4-bit quantizationQuantizationA compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.Learn more →.
SFT and RLHF in Practice
Supervised Fine-TuningSupervised Fine-TuningSupervised Fine-Tuning (SFT) is the process of training a pre-trained base model on curated instruction-response pairs to teach it specific tasks before preference alignment.Learn more → (SFT) trains the model on labeled promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →-response pairs: 'Given this prompt, the ideal response is...' This is straightforward and effective for style, format, and domain knowledge adaptation. The main challenge is dataset quality — poor examples teach poor behavior, and even a few bad examples can degrade a small fine-tune significantly.
RLHFRLHFReinforcement Learning from Human Feedback — a training technique that uses human preferences to teach AI models to be helpful, honest, and harmless.Learn more → (or its variant DPO — Direct Preference OptimizationDirect Preference OptimizationA training method that aligns language models with human preferences directly from preference pairs, bypassing the need for separate reward models or reinforcement learning.Learn more →) uses human (or AI) preference comparisons to shape behavior beyond what SFT can achieve. Rather than specifying the ideal response, you specify which of two responses is better. DPO has largely replaced standard RLHF for fine-tuning because it's more stable and doesn't require training a separate reward modelReward ModelA model trained on human preference data that scores AI outputs, enabling reinforcement learning systems to optimize for human-aligned responses.Learn more →.
Evaluating Your Fine-Tune
Evaluating fine-tuned models requires careful design. Automated metrics (accuracy, BLEUBLEUBLEU is a precision-based metric that evaluates machine translation quality by comparing n-gram overlap between generated text and reference translations.Learn more →, ROUGEROUGEROUGE is a recall-oriented evaluation metric that measures n-gram overlap between generated summaries and reference texts to assess summarization quality.Learn more →) are useful for structured tasks but miss quality dimensions like helpfulness, safety, and style. LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more →-as-judge evaluations (using GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4 or Claude to compare outputs) have become the standard approach for assessing generation quality at scale.
Always hold out a test set before fine-tuning — never evaluate on examples you trained on. Monitor for catastrophic forgettingCatastrophic ForgettingThe phenomenon where neural networks lose previously learned knowledge when trained on new tasks or data, requiring careful strategies to preserve existing capabilities.Learn more →: fine-tuning can degrade the model's general capabilities. Regularly run the model through a suite of general capability checks alongside your domain-specific evaluation to catch regressions early.