ROUGE.
ROUGE is a recall-oriented evaluation metric that measures n-gram overlap between generated summaries and reference texts to assess summarization quality.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of automatic evaluation metrics designed to assess the quality of text summarization systems by comparing generated summaries against human-written reference summaries. Unlike precision-focused metrics, ROUGE emphasizes recall by measuring how much of the reference content appears in the generated summary. This makes it particularly valuable for evaluating whether summarization models capture the essential information from source documents, which is critical for applications like news summarization, document analysis, and content generation.
ROUGE operates by calculating n-gram overlap statistics between candidate and reference texts, with several variants including ROUGE-N (unigram and bigram overlap), ROUGE-L (longest common subsequence), and ROUGE-S (skip-bigram co-occurrence). ROUGE-1 measures unigram overlap and captures basic content coverage, while ROUGE-2 focuses on bigram overlap to assess fluency and coherence. ROUGE-L evaluates sentence-level structure similarity without requiring consecutive matches, making it more flexible for different summary styles. Each variant produces precision, recall, and F1-score measurements, though recall scores are typically emphasized given the metric's design philosophy.
While ROUGE remains the standard evaluation metric for summarization tasks and is widely used in academic research and industry benchmarks, it has notable limitations that practitioners must understand. The metric focuses purely on lexical overlap and cannot assess semantic similarity, meaning paraphrased content that conveys the same meaning may receive low scores. ROUGE also struggles with creative or abstractive summaries that use different vocabulary than the reference text, potentially penalizing high-quality summaries that demonstrate true understanding. Modern evaluation approaches increasingly combine ROUGE with human evaluation and semantic similarity metrics to provide more comprehensive assessment of summarization quality.