Checkpoint.

A checkpoint is a complete snapshot of a machine learning model's parameters, optimizer state, and training metadata saved at a specific point during training. Checkpoints serve as recovery points that enable researchers and engineers to resume interrupted training sessions, evaluate model performance at different stages, or deploy models for inference. They are essential for managing long-running training processes that can take days or weeks to complete, providing insurance against hardware failures, power outages, or other interruptions that could otherwise result in losing significant computational work.

Checkpoints typically contain the model's learned weights, the optimizer's internal state (including momentum and learning rate schedules), the current epoch or step number, and often additional metadata like loss curves or validation metrics. Modern deep learning frameworks automatically save checkpoints at regular intervals, though practitioners can also trigger manual saves at key milestones. The frequency of checkpointing involves a trade-off between storage overhead and potential lost work—too frequent saves consume disk space and I/O bandwidth, while infrequent saves risk losing more progress if training is interrupted.

In practice, checkpoints enable several critical workflows beyond simple recovery. Researchers use them to compare model performance across training stages, implement early stopping when validation metrics plateau, and create model ensembles from different training points. For deployment, the final checkpoint becomes the production model, while intermediate checkpoints can serve as fallback versions. A common misconception is that checkpoints are just model weights—they actually preserve the entire training state, making them significantly larger than inference-only model files but essential for reproducible research and robust training pipelines.