Model Quantization: Running LLMs on Less Hardware

How quantization reduces model size and inference cost by using lower-precision numbers — making 70B parameter models run on a single GPU and enabling on-device AI.

Key takeaways
  • INT4 quantization reduces a 70B model from 140GB to 35GB, enabling single GPU deployment.
  • 4-bit quantization typically retains 95-99% of original model benchmark performance.
  • Enables fine-tuning 70B models on single 48GB GPUs by quantizing base weights to 4-bit.
  • Quantization powers AI on iPhone 15 Pro and Galaxy devices using 3B-8B parameter models.
  • Quantized on-device models are useful for bounded tasks but don't match frontier 70B capabilities.

What Quantization Is and Why It Matters

Neural network weights are normally stored as 32-bit (FP32) or 16-bit (FP16/BF16) floating-point numbers. A 70B parameter model in 16-bit precision requires 140GB of GPU memory — more than three high-end consumer GPUs. QuantizationQuantizationA compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.Learn more → reduces precision: INT8 (8-bit integers) halves that to 70GB; INT4 (4-bit) gets to 35GB — now feasible on a single professional GPU or 4× consumer GPUs.

The fundamental question is how much quality is lost by reducing numerical precision. Modern quantization research has shown that 4-bit quantization typically retains 95–99% of the original model's benchmarkBenchmarkA standardized test or set of tasks used to evaluate and compare the capabilities of different AI models on a common scale.Learn more → performance — a remarkable result that has made quantized models the standard for self-hosted deployments.

Key Quantization Methods

GPTQ (Generative Pre-Training Quantization) applies per-layer weight quantization using a small calibration dataset, minimizing reconstruction error. It typically produces the best quality for INT4 quantization of large models. AWQ (Activation-aware Weight Quantization) identifies and protects the most important weights from heavy quantization, further improving quality.

GGUFGGUFA file format for distributing quantized language models optimized for efficient local inference, popularized by llama.cpp.Learn more → (formerly GGML) is the dominant format for CPU-compatible quantization, used by llama.cpp. It supports a range of quantization levels from Q2_K (extreme compression) to Q8_0 (near-lossless). The Q4_K_M quantization level offers an excellent quality-size tradeoff for most use cases and is the recommended default for local deployment.

QLoRA: Quantized Fine-Tuning

QLoRA combines quantization with LoRALoRALow-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that trains small weight matrices while freezing the base model.Learn more → fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more →, enabling fine-tuning of large models on consumer hardware. The base model is loaded in 4-bit quantized form (frozen); only the LoRA adapter weights are trained in full precision. A 70B model that would normally require 160GB of GPU memory for fine-tuning can be fine-tuned on a single 48GB GPU with QLoRA.

QLoRA quality is surprisingly close to full fine-tuning. The 2023 paper that introduced it demonstrated that QLoRA fine-tunes of LLaMA matched or outperformed models fine-tuned with other methods at similar scales. This democratized fine-tuning, enabling practitioners without access to GPU clusters to adapt frontier-scale models.

On-Device LLMs

Quantization is what enables on-device AI. Apple's iPhone 15 Pro and M-series Macs run 3B–8B parameter quantized models natively. Samsung's Galaxy AI features use on-device quantized models for certain tasks. Microsoft's Phi-3 Mini at 3.8B parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → is designed to run on edge devices with 4-bit quantization.

On-device inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → eliminates network latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more →, preserves privacy (data never leaves the device), and enables offline operation. The trade-off is capability: a 7B quantized model is genuinely useful but doesn't match a frontier 70B model. The sweet spot for on-device is well-defined, bounded tasks where smaller models are adequate.

Read next
7 min read

Training vs Inference: Two Phases of an LLM's Life

Understand the difference between training an LLM (creating it) and inference (using it), including what happens at each stage, the costs involved, and why they matter for builders.

7 min read

Open-Weight vs Open-Source Models: What's the Difference?

Why 'open-source AI' is often a misleading term — and what it actually means when a model is open-weight, what's included, what's not, and why it matters for developers.

8 min read

Mixture of Experts: How LLMs Scale Efficiently

The architecture behind GPT-4, Llama 4, and Mistral — where only a subset of model parameters are active per token, enabling huge capacity at manageable inference cost.