Loss Function.

A loss function is the mathematical foundation of machine learning training, serving as the objective that algorithms minimize to improve model performance. It measures the discrepancy between predicted outputs and actual target values, providing a single numerical score that represents how 'wrong' the model currently is. Without loss functions, there would be no way to systematically guide a model toward better predictions, making them essential for any learning algorithm.

Different types of models use different loss functions depending on their task and output format. For classification problems, cross-entropy loss is common, while regression tasks often use mean squared error. Large language models specifically rely on next-token prediction cross-entropy loss, where the model learns to predict the probability distribution over the entire vocabulary for each position in a sequence. The gradient of this loss function with respect to model parameters determines how weights should be adjusted during backpropagation.

The choice of loss function significantly impacts what a model learns and how it behaves in practice. A well-designed loss function aligns the training objective with the desired real-world performance, while a poorly chosen one can lead to models that optimize for the wrong thing. Common misconceptions include thinking that lower loss always means better performance—overfitting can produce very low training loss but poor generalization. Additionally, the loss function used during training may differ from the metrics used to evaluate final model quality.