How to run Llama 3.3 70B on a workstation

Llama 3.3 70B is Meta's best dense open-source model, matching GPT-4o on many benchmarks. This guide covers hardware requirements, quantisation selection, and optimal settings for running it on a high-end workstation or Apple Silicon Mac.

Hardware requirements

At Q4_K_M quantisation, the Llama 3.3 70B model weighs around 42 GB. You need either: an Apple Silicon Mac with 48 GB+ of unified memory (M2 Max/Ultra, M3 Max/Ultra, M4 Max/Ultra), or a workstation with an NVIDIA GPU with 48 GB+ VRAMVRAMVideo Random Access Memory (VRAM) is the dedicated memory on graphics cards that stores model weights and the KV cache during LLM inference.Learn more → (RTX 6000 Ada, A100 80GB, dual RTX 3090/4090), or a system with 64 GB+ RAM if you are willing to run on CPU (much slower).

CPU inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → is feasible for non-interactive tasks. At Q4 on a modern desktop CPU with 64 GB RAM, you can expect 2–4 tokens per secondTokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.Learn more → — acceptable for batch processing but too slow for chat. Apple Silicon with 48 GB unified memory runs at 12–18 tok/s at Q4, which is comfortable for real-time use.

Download and run with Ollama

With 48 GB unified memory, run: `ollama pull llama3.3:70b`. The download is around 43 GB. Ollama automatically uses Metal acceleration on Apple Silicon and CUDA on NVIDIA GPUs.

Start a session: `ollama run llama3.3:70b`. On M2 Max (32 GPU cores, 48 GB), you will see around 14 tok/s for the 70B model — fast enough for comfortable interactive use. On M3 Max, expect 18–22 tok/s.

Choose the right quantisation

Q4_K_M is the recommended starting point — it reduces model size by ~75% versus BF16 with only a ~1–2% quality drop on most benchmarks. Q5_K_M improves quality slightly (0.5–1%) at 20% larger size — worth it if you have 52+ GB VRAM. Q8_0 is essentially lossless quality at about half the BF16 size but requires 70+ GB.

For the 70B model: Q4_K_M is ~42 GB (fits 48 GB), Q5_K_M is ~50 GB (fits 52 GB), Q8_0 is ~74 GB (requires 80 GB). If your system is right on the edge, go one notch down — running comfortably at Q4 beats thrashing memory at Q5.

Optimise inference speed

In Ollama, set the number of GPU layers to max: add `OLLAMA_NUM_GPU=99` to your environment (or set it in the Modelfile). On Apple Silicon this is handled automatically — all layers go to the GPU.

Reduce context length for shorter conversations. Llama 3.3 supports 128K context, but loading a 128K KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → uses significant memory. Set context to 8K for most tasks: `ollama run llama3.3:70b --ctx-size 8192`. This alone can increase throughputThroughputThe number of tokens or requests an AI system can process per unit time, measuring overall capacity rather than per-request speed.Learn more → by 20–30% on memory-constrained systems.

Batch multiple prompts together when processing documents. Llama 3.3's promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → processing (prefillPrefillThe initial inference phase where an LLM processes the entire input prompt in parallel to populate the KV cache before beginning sequential token generation.Learn more →) is very fast — it is the generation (decoding) step that is slow. Pre-processing 10 documents into a queue and letting the model decode continuously is more efficient than sending one prompt at a time.