Gradient Accumulation.

Gradient accumulation is a memory optimization technique used during neural network training that allows practitioners to simulate larger effective batch sizes without exceeding hardware memory limitations. Instead of updating model parameters after each mini-batch, the technique accumulates (sums) gradients across several mini-batches before performing a single parameter update. This approach is particularly crucial for training large language models where memory constraints often prevent using optimal batch sizes directly.

The process works by computing gradients for each mini-batch as usual, but instead of immediately applying these gradients to update parameters, they are stored and summed with gradients from subsequent mini-batches. Once the desired number of accumulation steps is reached, the accumulated gradients are averaged and used for a single parameter update, after which the gradient accumulator is reset. This creates mathematically equivalent training dynamics to using a larger batch size, though with slightly different computational characteristics due to the sequential processing of mini-batches.

Gradient accumulation enables training with effective batch sizes that would otherwise be impossible due to GPU memory limitations, often improving training stability and convergence for large models. However, it comes with trade-offs including increased training time per effective batch and potential differences in batch normalization behavior. A common misconception is that gradient accumulation always improves training quality—while it can help achieve better batch size targets, the optimal accumulation strategy depends on the specific model architecture, dataset, and available computational resources.