What quantisation does
A full-precision (BF16) LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → stores each model weight as a 16-bit float. At this precision, a 7B parameter model occupies 14 GB. Quantisation reduces the number of bits used per weight — Q4 quantisation stores each weight in about 4.5 bits, shrinking that 7B model to roughly 4 GB.
The tradeoff is quality: fewer bits means less precision, which introduces small errors in each computation. For most real-world use cases, Q4 and Q5 quantisations are imperceptible in output quality. Q2 and Q3 cause noticeable degradation, especially on reasoning tasks.
The quantisation levels explained
Q2_K: 2–3 bits per weight. Smallest possible size — a 7B model fits in ~3 GB. Significant quality loss, not recommended for tasks requiring precision. Use only if you have very limited VRAMVRAMVideo Random Access Memory (VRAM) is the dedicated memory on graphics cards that stores model weights and the KV cache during LLM inference.Learn more → and need basic text generation.
Q3_K_M / Q3_K_L: 3–4 bits. Still noticeably degraded vs full precision, but usable for casual tasks. A 7B model is ~3.5 GB.
Q4_K_M (recommended default): ~4.5 bits. The sweet spot. Quality loss is minimal — typically 1–3% on standard benchmarks. A 7B model is ~4.1 GB. This is what Ollama uses when you pull a model without specifying a quantisation.
Q5_K_M: ~5.5 bits. Higher quality, ~20% larger than Q4_K_M. Choose this when you have headroom and care about marginal quality (e.g. creative writing, nuanced analysis).
Q6_K: ~6.5 bits. Approaching Q8 quality. Good for 13B+ models where you want near-lossless quality but not the size of Q8.
Q8_0: 8 bits. Essentially lossless — indistinguishable from BF16 in practice. A 7B model is ~7.7 GB. Use when accuracy is critical and VRAM is not a constraint.
Matching quantisation to VRAM
4 GB VRAM: 3B models at Q4–Q8, 7B models at Q2 (with significant quality loss). Consider a 3B model like Phi-4 Mini instead of a degraded 7B.
6 GB VRAM: 7B models at Q4_K_M, 3B models at Q8. This is the most common entry-level gaming GPU scenario.
8 GB VRAM: 7B models at Q6–Q8, or 14B models at Q2–Q3 (with quality loss). Recommended: 7B at Q8 or 14B at Q4 with a few layers on CPU.
12 GB VRAM: 14B models at Q4_K_M, or 7B models at any quantisation. This is the sweet spot for the RTX 3060 12 GB and RTX 4070.
16 GB VRAM: 14B at Q6, or 20–24B models at Q4. Apple M1/M2 16 GB unified memory falls here.
24 GB VRAM: 33B models at Q4, or 20B at Q6–Q8.
48 GB (unified): 70B models at Q4_K_M. M2 Max / M3 Max sweet spot.
80 GB VRAM: 70B models at Q8, or 30B models at BF16.
The _K suffix explained
The K variants (Q4_K_M, Q5_K_S, etc.) use k-quants, a more sophisticated quantisation method that assigns fewer bits to less important layers. _S is small (faster), _M is medium (better quality), _L is large (best quality in the group). For most use cases, _M is the right choice.
When in doubt, pick the highest _K_M that fits comfortably in your VRAM with a few GB to spare for the KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more →. A model that fits with 1 GB to spare will be slower than one with 3 GB spare because it will need to page the KV cache.