Perplexity.
A statistical measure of how well a language model predicts text — lower perplexity means the model finds the text less surprising and is a better predictor.
Perplexity is a fundamental evaluation metric for language models derived from information theory. It measures how 'perplexed' or surprised a model is by a test dataset. A lower perplexity score means the model assigns higher probability to the actual next words in a sequence, indicating better language modeling.
Perplexity is calculated as the exponentiated cross-entropy loss on held-out test data. A perplexity of 10 means the model is as uncertain as if it had to choose uniformly among 10 equally likely options at each step. Perplexity is useful for comparing models trained on the same type of data but less useful across domains.
While perplexity is important for model researchers, it has limitations as a user-facing metric. A model can have low perplexity but still hallucinate facts, follow instructions poorly, or generate unsafe content. This is why the field has moved toward task-specific benchmarks and human preference evaluation rather than relying solely on perplexity.