What is Ollama and why use it
Ollama is an open-source tool that lets you download, manage, and run large language models on your own machine with a single terminal command. It handles all the complexity of GGUFGGUFA file format for distributing quantized language models optimized for efficient local inference, popularized by llama.cpp.Learn more → quantisation, llama.cpp compilation, and model management so you don't have to.
Once installed, running a model is as simple as `ollama run llama3.3`. Models download automatically on first run and are cached locally for offline use. Ollama also exposes a local HTTP server at port 11434 that is compatible with the OpenAI API, so you can point any OpenAI SDK client at it.
Install Ollama
On macOS, download the Ollama app from ollama.com and drag it to your Applications folder. The menu-bar icon confirms it is running. Alternatively install via Homebrew: `brew install ollama`.
On Linux, run the official one-liner: `curl -fsSL https://ollama.com/install.sh | sh`. This installs the `ollama` binary and registers a systemd service that starts automatically on boot.
On Windows, download the installer from ollama.com. It installs as a background service and adds `ollama` to your PATH so you can use it from PowerShell or Command PromptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →.
Pull and run your first model
Open a terminal and run `ollama pull phi4-mini` to download Microsoft's 3.8B model — it is under 2.5 GB and runs comfortably on 4 GB of RAM. For something more capable on 8 GB RAM, try `ollama pull deepseek-r1:7b`.
Once downloaded, start an interactive chat session with `ollama run phi4-mini`. Type your message and press Enter. Type `/bye` to exit the session. To run a quick one-shotOne-ShotA prompting technique where exactly one example is provided to demonstrate the desired task format before presenting the actual query.Learn more → query without entering interactive mode, use: `ollama run phi4-mini 'Explain transformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more → attention in one paragraph'`.
Use the REST API
Ollama runs a local server at `http://localhost:11434`. You can call it from any HTTP client: `curl http://localhost:11434/api/generate -d '{"model": "phi4-mini", "prompt": "Why is the sky blue?", "stream": false}'`.
Because the server exposes an OpenAI-compatible `/v1/chat/completions` endpoint, you can reuse any existing OpenAI SDK code by changing the base URL: `client = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')`. This makes it trivial to switch between local and cloud models during development.
Manage your models
Use `ollama list` to see all downloaded models with their sizes, `ollama rm <model>` to delete one, and `ollama ps` to see which models are currently loaded in memory. Ollama keeps a model warm in GPU/memory for a few minutes after use to speed up subsequent requests.
To see available models, visit the Ollama model library at ollama.com/library. It lists hundreds of open-source models with their sizes, quantisation levels, and recommended hardware.