Why vLLM for production
HuggingFace Transformers is the standard for model loading and fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more →, but its inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → is slow under load because each request allocates KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → memory statically. vLLM's PagedAttentionPagedAttentionA memory management technique from vLLM that allocates attention cache in pages like an operating system, reducing memory waste and enabling higher concurrency.Learn more → manages the KV cache like virtual memory — dynamically allocating pages across requests. This allows continuous batchingContinuous BatchingA serving optimization technique that dynamically adds and removes requests from GPU batches during execution to maximize throughput and hardware utilization.Learn more → of hundreds of concurrent requests on a single GPU.
For a 7B model on an A100, vLLM achieves around 3000–4000 tokens/second throughputThroughputThe number of tokens or requests an AI system can process per unit time, measuring overall capacity rather than per-request speed.Learn more → — 10–20× more than a naive HuggingFace setup. For APIs serving multiple users, this dramatically reduces cost per query.
Install vLLM
vLLM requires an NVIDIA GPU with CUDA 12.1+. Install via pip: `pip install vllm`. The package is large (~2 GB) as it includes compiled CUDA kernels. For the latest features, install from source: `pip install git+https://github.com/vllm-project/vllm.git`.
On AMD GPUs (MI250/MI300), use the ROCm build: `pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.1`. Apple Silicon is not supported — use Ollama or llama.cpp for local Mac inference.
Serve your first model
Start the server: `vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --port 8000`. The `--tensor-parallel-size 2` flag splits the model across 2 GPUs. `--gpu-memory-utilization 0.9` allows vLLM to use 90% of VRAMVRAMVideo Random Access Memory (VRAM) is the dedicated memory on graphics cards that stores model weights and the KV cache during LLM inference.Learn more → for the KV cache.
The server exposes an OpenAI-compatible API at `http://localhost:8000/v1`. Any OpenAI SDK client works without modification. The `/v1/models` endpoint lists the available model. The model name in requests should match the HuggingFace model ID.
Configure for throughput vs latency
For maximum throughput (batch processing): increase `--max-num-seqs 256` (max concurrent sequences) and enable continuous batching (default). Set `--max-num-batched-tokens 16384` to allow larger batches. Throughput improves linearly with batch sizeBatch SizeThe number of training examples processed together in a single forward and backward pass before updating model parameters.Learn more → up to GPU saturation.
For minimum latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → (interactive chat): use a smaller `--max-num-seqs 8`, enable speculative decodingSpeculative DecodingA technique where a small draft model proposes multiple tokens that a larger model verifies in parallel, maintaining identical output while reducing generation latency.Learn more → with `--speculative-model <small-draft-model>`, and use `--enforce-eager` to skip CUDA graph capture for faster startup. Speculative decoding reduces latency by 2–3× for chat-length generations.
Quantised and large model serving
For a 70B model on a single 80GB GPU, use AWQ quantisation: `vllm serve meta-llama/Meta-Llama-3-70B-Instruct-AWQ --quantizationQuantizationA compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.Learn more → awq`. AWQ (Activation-aware Weight Quantisation) reduces model size to ~4 bits with minimal quality loss and is faster than GPTQ.
For models larger than your VRAM, use pipeline parallelism across multiple nodes: `--pipeline-parallel-size 2` splits the model's layers across 2 nodes. Combined with tensor parallelismTensor ParallelismA parallelization technique that splits individual weight matrices across multiple GPUs, enabling deployment of models too large for a single device.Learn more → (`--tensor-parallel-size 4` per node), you can serve 405B+ models across a GPU cluster.