Mixed Precision.

Mixed precision is a computational optimization technique that combines different floating-point precisions during neural network training and inference. Instead of using the traditional 32-bit floating-point (fp32) format throughout the entire process, mixed precision strategically employs 16-bit formats like half-precision (fp16) or brain floating-point (bf16) for most operations while keeping fp32 for critical computations that require higher numerical stability. This approach has become essential for training large language models efficiently, as it can reduce memory consumption by up to 50% and significantly accelerate training speed on modern GPUs with tensor cores.

The technique works by automatically identifying which operations can safely use lower precision without compromising numerical stability or final model quality. Most forward pass computations, gradient calculations, and weight updates can be performed in fp16 or bf16, while operations prone to numerical instability like loss scaling and certain normalizations remain in fp32. Modern frameworks like PyTorch and TensorFlow provide automatic mixed precision (AMP) implementations that handle the precision switching transparently. The key distinction between fp16 and bf16 lies in their range and precision trade-offs: fp16 offers higher precision but a smaller numerical range, while bf16 sacrifices some precision for a much larger range that matches fp32.

In practice, mixed precision enables training larger models on the same hardware or training existing models faster with minimal accuracy loss. However, it requires careful implementation to avoid gradient underflow in fp16 or ensure proper loss scaling. Common misconceptions include thinking that mixed precision always halves memory usage (actual savings depend on model architecture and implementation) or that it works equally well across all model types and sizes. While mixed precision has become standard for large model training, developers must monitor for numerical instabilities and may need to adjust learning rates or implement gradient clipping to maintain training stability.