Frequency Penalty.

Frequency penalty is a text generation parameter that systematically reduces the likelihood of selecting tokens (words or subwords) proportional to how many times they have already appeared in the current output sequence. This mechanism helps combat repetitive text generation, a common issue where language models get stuck in loops or overuse certain phrases. By penalizing previously used tokens, frequency penalty encourages models to explore more diverse vocabulary and produce more varied, natural-sounding text.

The parameter works by tracking token usage throughout generation and applying a multiplicative penalty to the logits (pre-softmax probabilities) of tokens that have appeared before. Unlike presence penalty which applies a fixed reduction regardless of usage count, frequency penalty scales the reduction based on actual occurrence frequency. The penalty is typically applied as a negative value subtracted from or multiplied against the token's original probability, with higher frequency penalties leading to stronger discouragement of repetition.

Frequency penalty proves particularly valuable for creative writing, content generation, and conversational AI where repetition feels unnatural to users. However, setting the penalty too high can lead to incoherent text as the model avoids necessary repetitions of common words like articles and prepositions. The optimal frequency penalty value depends on the specific use case, with technical documentation often requiring lower penalties than creative fiction to maintain clarity and precision.