Quantization.
A compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.
Quantization reduces the numerical precision of model weights from 32-bit or 16-bit floating point to lower-precision formats like 8-bit integers (INT8) or 4-bit integers (INT4). A model quantized to 4-bit uses roughly 4x less memory than its 16-bit equivalent, enabling larger models to run on consumer hardware.
Modern quantization methods like GPTQ, AWQ, and GGUF have become sophisticated enough that 4-bit quantized models typically retain over 95% of the original model's performance on benchmarks. The most accessible format is GGUF (used by llama.cpp), which allows running quantized versions of Llama, Mistral, and other open models on a laptop CPU.
Quantization is one of the most important techniques for democratizing access to powerful AI models. It is what allows a 70B parameter model — which would normally require 8 professional-grade GPUs — to run on a single consumer GPU with 24GB of VRAM, or even on a high-end laptop using the CPU.