Layer Normalization.

Layer normalization is a crucial technique in deep learning that normalizes the inputs to each layer by computing the mean and variance across the feature dimension for each training example independently. Unlike batch normalization which normalizes across the batch dimension, layer normalization operates on individual samples, making it particularly effective for sequence models and transformers where batch sizes may vary. This normalization helps address the internal covariate shift problem, where the distribution of layer inputs changes during training, leading to more stable and faster convergence.

The technique works by computing statistics (mean and standard deviation) across all features in a layer for each individual sample, then normalizing these activations to have zero mean and unit variance before applying learnable scale and shift parameters. In transformer architectures, layer normalization appears in two main configurations: pre-norm (applied before attention and feed-forward blocks) and post-norm (applied after these blocks). Pre-norm has become more popular in recent models as it tends to be more stable during training and allows for better gradient flow, while post-norm was used in earlier transformer implementations.

Layer normalization significantly improves training stability by reducing sensitivity to initialization and learning rates, often allowing for higher learning rates and faster convergence. However, it does add computational overhead during both training and inference, and the choice between pre-norm and post-norm configurations can affect model performance and training dynamics. A common misconception is that layer normalization always improves model performance - while it generally stabilizes training, the optimal normalization strategy depends on the specific architecture and task, with some recent research exploring alternatives like RMSNorm for even better efficiency.