Supervised Fine-Tuning.

Supervised Fine-Tuning (SFT) is a critical training phase where a pre-trained base language model is further trained on carefully curated datasets of input-output examples, typically instruction-response pairs. This process transforms a raw base model that primarily predicts the next token into one that can follow instructions and perform specific tasks. SFT serves as the foundation for creating useful AI assistants, occurring before more advanced alignment techniques like reinforcement learning from human feedback.

During SFT, the model learns from high-quality demonstrations where each training example consists of an instruction or prompt paired with a desired response. The training process uses standard supervised learning techniques, adjusting the model's parameters to maximize the likelihood of producing the correct outputs given the inputs. Unlike the broad, unsupervised pre-training phase, SFT focuses the model's capabilities on following instructions, answering questions, and completing tasks in a helpful manner. The quality and diversity of the SFT dataset directly impacts the model's ability to generalize to new instructions.

SFT represents a crucial bridge between raw language modeling capabilities and practical AI applications, but it has important limitations. Models trained only with SFT may still produce outputs that are technically correct but not aligned with human preferences for helpfulness, harmlessness, or honesty. This is why SFT is typically followed by preference alignment methods like RLHF. Additionally, SFT requires substantial computational resources and high-quality training data, making the curation of instruction datasets a critical bottleneck in developing capable AI systems.