Logits.

Logits are the raw numerical outputs that neural networks produce before any normalization, representing the model's confidence or preference for each possible token in its vocabulary. In language models, logits serve as the foundation for text generation, with each position in the logits vector corresponding to a specific token the model could generate next. These unnormalized scores are crucial because they contain the model's pure assessment of likelihood before any mathematical transformation shapes the final probability distribution.

The transformation from logits to probabilities occurs through the softmax function, which exponentiates each logit value and normalizes them so they sum to 1.0. This process preserves the relative ordering of preferences while creating a valid probability distribution. Logits can be any real number, including negative values, but after softmax transformation, all probabilities fall between 0 and 1. The magnitude of differences between logits determines how confident or uncertain the model appears in its token selection.

Understanding logits enables fine-grained control over model behavior through techniques like temperature scaling, top-k filtering, and logit bias adjustments. Developers can manipulate logits directly to encourage or discourage specific tokens, implement custom sampling strategies, or analyze model confidence patterns. A common misconception is that higher logits always mean better predictions, but the relative differences between logits matter more than absolute values, and extremely high logits can indicate overconfident or poorly calibrated models.