Dropout.

Dropout is a fundamental regularization technique in neural networks that addresses overfitting by randomly setting a fraction of neuron activations to zero during each training step. This forces the network to learn more robust representations that don't depend too heavily on any single neuron or pathway. By preventing co-adaptation between neurons, dropout encourages the model to develop multiple independent ways to recognize patterns, leading to better generalization on unseen data.

During training, dropout randomly selects neurons according to a specified probability (typically 0.2 to 0.5) and temporarily removes them from the network by setting their outputs to zero. The remaining active neurons must compensate for the missing information, which prevents the formation of complex co-dependencies that might work well on training data but fail on new examples. Importantly, dropout is only applied during training - during inference, all neurons are active but their outputs are scaled to account for the training-time dropout rate.

While dropout significantly improves generalization in many neural architectures, it comes with trade-offs including longer training times and the need to tune the dropout rate for optimal performance. Modern architectures like Transformers often use alternative regularization methods such as layer normalization and attention dropout. A common misconception is that higher dropout rates always lead to better regularization - in practice, too much dropout can prevent the network from learning effectively, while too little may not provide sufficient regularization benefits.