Weight Decay.

Weight decay is a regularization method that adds a penalty term to the loss function proportional to the squared magnitude of the model's weights, effectively encouraging smaller weight values throughout training. This technique helps prevent overfitting by discouraging the model from relying too heavily on any single parameter, forcing it to learn more generalizable patterns rather than memorizing training data. Weight decay is particularly crucial in deep learning where models have millions or billions of parameters that can easily overfit to training examples.

The mechanism works by adding a term λ||w||² to the loss function, where λ is the weight decay coefficient and w represents the model weights. During each optimization step, this penalty causes weights to shrink toward zero unless they contribute significantly to reducing the primary loss. Weight decay differs from L2 regularization in subtle but important ways when combined with adaptive optimizers like Adam - leading to the development of AdamW, which decouples weight decay from the gradient-based updates to achieve more effective regularization.

In practice, weight decay acts as a form of implicit model selection, automatically reducing model complexity during training rather than requiring manual architecture changes. The weight decay coefficient typically ranges from 1e-6 to 1e-1, with larger values providing stronger regularization but potentially underfitting the model. A common misconception is that weight decay and L2 regularization are identical - while mathematically equivalent for SGD, they behave differently with adaptive optimizers, making proper weight decay implementation crucial for achieving optimal model performance.