Cross-Entropy.

Cross-entropy is the fundamental loss function used to train large language models, quantifying how well a model's predicted probability distribution over tokens matches the true next token in a sequence. It penalizes confident wrong predictions more heavily than uncertain ones, making it ideal for next-token prediction tasks where models must learn to assign high probability to correct continuations. This mathematical framework enables models to learn the statistical patterns of language by minimizing the gap between their predictions and reality.

The cross-entropy loss works by comparing the model's output probability distribution across the entire vocabulary with a one-hot encoded target vector representing the correct token. When the model assigns high probability to the correct token, the loss is low; when it confidently predicts the wrong token, the loss spikes dramatically due to the logarithmic nature of the calculation. This asymmetric penalty structure encourages models to be both accurate and appropriately uncertain, leading to better calibrated predictions during inference.

In practice, cross-entropy loss drives the core learning process during both pre-training and fine-tuning phases of language model development. While effective for teaching models to predict text, it can sometimes lead to overconfident predictions or difficulty with rare tokens that appear infrequently in training data. Understanding cross-entropy helps explain why language models sometimes struggle with edge cases or produce hallucinations when they confidently predict tokens that seem plausible but are factually incorrect.