Latency.
The delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).
In the context of LLM APIs, latency typically refers to the time from submitting a request to when text begins streaming back. This is called Time to First Token (TTFT) and is a critical metric for real-time applications like chatbots, coding assistants, and voice interfaces where users experience the delay as unresponsiveness.
Latency is influenced by model size, infrastructure load, prompt length, and inference hardware. Smaller models like Claude Haiku and Gemini Flash have substantially lower latency than frontier models like GPT-5 or Claude Opus. Specialized inference providers like Groq using LPU hardware achieve extremely low latency on smaller open models.
There is an inherent tension between latency and capability. The most capable models are large and slow; the fastest models sacrifice some capability for speed. Streaming responses mitigate perceived latency by showing tokens as they arrive rather than waiting for the complete response, which is why most modern AI interfaces use streaming by default.
Related terms.
- Time to First TokenTime to First Token (TTFT) measures the latency from sending a request to receiving the first streamed token, primarily determined by the prefill phase.
- Tokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.