Epoch.

An epoch represents one complete iteration through an entire training dataset during the machine learning training process. In the context of large language models, this means the model has seen and processed every piece of text in its training corpus exactly once. The concept is fundamental to understanding training dynamics, as it provides a standardized way to measure training progress and compare different training runs across models of varying sizes and datasets.

During each epoch, the model processes data in smaller chunks called batches, updating its parameters after each batch to gradually improve its performance. The number of epochs required for optimal training varies significantly based on model architecture, dataset size, and training objectives. Interestingly, many modern large language models are trained for less than one full epoch over their massive datasets, as the sheer volume of training data means that even a partial pass can provide sufficient learning signal.

The choice of epoch count involves critical trade-offs between model performance and computational cost, with too few epochs leading to underfitting and too many causing overfitting. For large language models trained on internet-scale datasets, practitioners often monitor validation metrics rather than completing predetermined epoch counts, stopping training when performance plateaus. This approach reflects the practical reality that with datasets containing trillions of tokens, even a fraction of an epoch represents enormous computational investment and learning opportunity.