BLEU.

BLEU (Bilingual Evaluation Understudy) is a widely-used automatic evaluation metric for machine translation that measures the quality of generated text by comparing it to one or more reference translations. Developed in the early 2000s, BLEU became the dominant standard for evaluating translation systems because it provides a fast, reproducible way to assess translation quality without human evaluators. The metric produces scores between 0 and 1 (often multiplied by 100), where higher scores indicate better alignment with reference translations.

BLEU works by calculating precision scores for n-grams (sequences of n consecutive words) of various lengths, typically 1-grams through 4-grams, between the candidate translation and reference texts. It combines these precision scores using a geometric mean and applies a brevity penalty to prevent artificially high scores from very short translations. The metric focuses purely on precision—measuring how many n-grams in the generated text appear in the references—but does not account for recall, meaning it doesn't penalize missing content as long as what is present matches the references.

While BLEU remains standard in machine translation research, it has significant limitations that practitioners must understand. The metric correlates poorly with human judgment for individual sentences and can be gamed by systems that produce short, safe translations with high n-gram overlap but poor semantic quality. BLEU also struggles with paraphrases and synonyms, potentially penalizing perfectly valid translations that use different wording than the references. Modern evaluation increasingly supplements BLEU with other metrics like BERT-based scores, human evaluation, or task-specific measures, especially as language models move beyond translation to more creative and diverse text generation tasks.