Why llama.cpp over Ollama for serving
Ollama is the easiest local model runner but has limited configurability. llama.cpp's `llama-server` exposes more fine-grained control: parallel request processing with continuous batchingContinuous BatchingA serving optimization technique that dynamically adds and removes requests from GPU batches during execution to maximize throughput and hardware utilization.Learn more →, speculative decodingSpeculative DecodingA technique where a small draft model proposes multiple tokens that a larger model verifies in parallel, maintaining identical output while reducing generation latency.Learn more → with a draft model, detailed per-request metrics, and GGUFGGUFA file format for distributing quantized language models optimized for efficient local inference, popularized by llama.cpp.Learn more → model loading with full quantisation control.
For team deployments where multiple users will be hitting the same server simultaneously, llama.cpp's continuous batching feature is critical — it batches incoming requests and processes them together, significantly improving GPU utilisation.
Build llama.cpp with GPU support
Clone the repository: `git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp`.
For NVIDIA (CUDA): `cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j`. Requires CUDA toolkit 12.x installed.
For Apple Silicon (Metal): `cmake -B build -DGGML_METAL=ON && cmake --build build --config Release -j`. Metal is the default on macOS; this explicitly enables it.
For CPU-only: `cmake -B build && cmake --build build --config Release -j`. Much slower but works on any hardware.
Start the server
Download a GGUF model (from Hugging Face) and start the server: `./build/bin/llama-server -m models/phi-4-q4_k_m.gguf --host 0.0.0.0 --port 8080 -ngl 99 -c 8192 --parallel 4`.
`-ngl 99` offloads all layers to GPU. `--parallel 4` allows 4 simultaneous requests with continuous batching. `-c 8192` sets the context per slot. Adjust `--parallel` based on your VRAMVRAMVideo Random Access Memory (VRAM) is the dedicated memory on graphics cards that stores model weights and the KV cache during LLM inference.Learn more → — each parallel slot requires the full KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more → allocation.
Connect clients
The server exposes `/v1/chat/completions`, `/v1/completions`, and `/v1/models` — compatible with any OpenAI SDK client. Point your client at `http://your-server:8080/v1` with any API key.
For team access on a local network, bind to `--host 0.0.0.0` and share the server's IP with your team. Add nginx as a reverse proxy for HTTPS if accessing over the public internet. Add basic auth to the nginx config to prevent unauthorised access.
Enable speculative decoding
Speculative decoding uses a small draft model to propose multiple tokens at once, which the larger model then verifies in parallel. This can increase throughputThroughputThe number of tokens or requests an AI system can process per unit time, measuring overall capacity rather than per-request speed.Learn more → by 2–3× for long generations. Use a model from the same family as your main model: a 3B draft model for a 70B main model.
Start the server with speculative decoding: `./build/bin/llama-server -m models/llama-70b-q4.gguf --draft models/llama-3b-q4.gguf --n-draft 8 ...`. The `--n-draft 8` flag specifies how many draft tokens to generate and verify per step.