How to evaluate a fine-tuned model against a baseline

Knowing when your fine-tuned model is actually better than the base model requires systematic evaluation. This guide covers benchmark datasets, LLM-as-judge evaluation, and metrics for task-specific assessment.

Define success before you train

Before training, define what 'better' means for your task. Is it lower perplexityPerplexityA statistical measure of how well a language model predicts text — lower perplexity means the model finds the text less surprising and is a better predictor.Learn more → on your validation set? Higher accuracy on specific test cases? Better human preference ratings? Without a clear definition, you cannot know if fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more → helped.

Create a held-out test set of 50–200 examples that the model will never train on. This is your ground truthGround TruthThe reference correct answer used to score model outputs during evaluation, serving as the objective standard against which AI performance is measured.Learn more → for evaluation. Run the base model on it before fine-tuning to establish a baseline.

Automatic evaluation metrics

For classification tasks: accuracy, F1 scoreF1 ScoreThe harmonic mean of precision and recall, providing a single balanced metric that equally weighs both false positives and false negatives in classification tasks.Learn more →, and confusion matrix. These are unambiguous and fully automated.

For generation tasks: ROUGEROUGEROUGE is a recall-oriented evaluation metric that measures n-gram overlap between generated summaries and reference texts to assess summarization quality.Learn more → (overlap with reference text) and BLEUBLEUBLEU is a precision-based metric that evaluates machine translation quality by comparing n-gram overlap between generated text and reference translations.Learn more → are traditional but correlate poorly with human judgment. BERTScore (semantic similarity) is a better choice for tasks where there are multiple valid phrasings.

Perplexity on your validation set is a fast sanity check: lower is better and signals the model has learned the target distribution.

LLM-as-judge evaluation

The most useful evaluation for generation quality is using a strong LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → (GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o or Claude Opus) to compare your fine-tuned model's outputs against the base model, side by side. Present both outputs without labels (blind) and ask the judge: 'Which response better answers the question and follows the correct format? Respond with A or B and a one-sentence reason.'

Run this comparison on 50–100 test cases and compute the win rateWin RateThe percentage of head-to-head comparisons where one AI model is judged superior to another by human evaluators or automated systems.Learn more →: what percentage of the time did the fine-tuned model win? A win rate above 65% indicates a meaningful improvement. Below 55% suggests the fine-tuning had minimal effect or may have caused regression.

Check for regressions

Fine-tuning for a specific task can degrade general capabilities — this is called catastrophic forgettingCatastrophic ForgettingThe phenomenon where neural networks lose previously learned knowledge when trained on new tasks or data, requiring careful strategies to preserve existing capabilities.Learn more →. Always test the fine-tuned model on a few general-knowledge and instruction-following examples that are unrelated to the fine-tuning task.

Use Eleuther AI's `lm-evaluation-harness` to run standard benchmarks on both the base and fine-tuned models: `python -m lm_eval --model hf --model_args pretrained=my-fine-tuned-model --tasks mmluMMLUA comprehensive benchmark evaluating language models across 57 academic subjects from elementary to professional level using multiple-choice questions.Learn more →,hellaswagHellaSwagA commonsense reasoning benchmark where AI models must select the most plausible continuation from four options for everyday scenarios.Learn more →,arc_easy`. Any benchmarkBenchmarkA standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.Learn more → drop of more than 5% suggests the fine-tuning was too aggressive.