Softmax.

The softmax function is a fundamental component in neural networks that transforms a vector of arbitrary real numbers (called logits) into a probability distribution where all values sum to 1.0. This transformation is crucial for tasks like classification and language modeling, where models need to output probabilities over different possible outcomes. The function ensures that higher logits correspond to higher probabilities while maintaining mathematical properties that make gradient-based optimization feasible.

Softmax works by exponentiating each logit value and then normalizing by the sum of all exponentiated values, creating a smooth, differentiable mapping from raw scores to probabilities. The temperature parameter scales the logits before applying the exponential function: lower temperatures make the distribution sharper (more confident), while higher temperatures make it flatter (more uniform). At temperature 1.0, the standard softmax is applied, while temperatures approaching zero create near-deterministic outputs and very high temperatures approach uniform distributions.

In language models, softmax is applied to the final layer's logits to determine the probability of each possible next token, making temperature a key hyperparameter for controlling output randomness and creativity. A common misconception is that softmax probabilities represent true confidence levels, but they can be overconfident and don't always reflect genuine uncertainty. The function's exponential nature also makes it sensitive to outliers, where one very large logit can dominate the entire distribution, leading to techniques like top-k and nucleus sampling to manage this behavior.