FLOPs.

FLOPs (floating-point operations per second) represent the fundamental unit for measuring computational intensity in machine learning and AI systems. This metric quantifies how many mathematical operations involving decimal numbers a system can perform each second, making it essential for understanding the computational requirements of training large language models and running inference. FLOPs serve as a hardware-agnostic way to compare the computational demands of different models and training procedures, helping researchers and engineers estimate costs, time requirements, and infrastructure needs.

The measurement encompasses various mathematical operations including addition, subtraction, multiplication, and division of floating-point numbers, with modern AI workloads typically measured in teraFLOPs (trillions of operations per second) or petaFLOPs (quadrillions per second). Training large language models can require enormous FLOP counts, often measured in the range of 10^23 to 10^25 FLOPs for state-of-the-art models. Different hardware architectures like GPUs, TPUs, and specialized AI chips are optimized for different types of floating-point operations, with some excelling at lower-precision calculations that can dramatically increase FLOP throughput while maintaining model quality.

Understanding FLOPs helps organizations make informed decisions about hardware procurement, training schedules, and model architecture choices, as higher FLOP requirements directly translate to increased energy consumption, longer training times, and higher costs. However, FLOPs alone don't tell the complete story of model efficiency, as factors like memory bandwidth, data movement, and algorithmic optimizations can significantly impact real-world performance. Modern techniques like mixed-precision training, model quantization, and efficient architectures aim to reduce FLOP requirements while maintaining model capabilities, making FLOP efficiency a key consideration in sustainable AI development.