- LLM latency splits into time to first token (TTFT) and tokens per second (TPS) with different drivers.
- Time to first token determines perceived responsiveness more than generation speed for user experience.
- Longer prompts take proportionally longer to process, with 10,000-token prompts taking roughly 10x longer than 1,000-token prompts.
- Larger models generate fewer tokens per second than smaller models on the same hardware due to increased computation.
- Mixture-of-Experts models like Mixtral achieve large model quality at smaller inference costs by activating only a fraction of parameters.
- Groq's LPU architecture achieves 10-20x faster TPS than GPU-based inference by eliminating memory bandwidth bottlenecks.
Two Metrics That Define LLM Latency
LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → is not a single number — it splits into two distinct metrics with different drivers. Time to first tokenTime to First TokenTime to First Token (TTFT) measures the latency from sending a request to receiving the first streamed token, primarily determined by the prefill phase.Learn more → (TTFT) measures how long from sending a request until the first character of output appears. Tokens per secondTokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.Learn more → (TPS) measures how quickly the model generates subsequent output. Both matter, but for different reasons.
TTFT determines perceived responsiveness — users feel an LLM as 'snappy' or 'laggy' based on this number more than generation speed. TPS determines how quickly long responses complete. For chat interfaces, a fast TTFT with moderate TPS feels much better than slow TTFT with fast TPS, because streamingStreamingStreaming returns a model's tokens incrementally as they're generated rather than waiting for the complete response to finish.Learn more → text feels engaging while waiting for the first word feels like the application is frozen.
What Drives Time to First Token
TTFT is dominated by the prefillPrefillThe initial inference phase where an LLM processes the entire input prompt in parallel to populate the KV cache before beginning sequential token generation.Learn more → phase of inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more →: processing the input promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → through the model. Longer prompts take longer to prefill. A 10,000-tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → prompt takes roughly 10x longer to process than a 1,000-token prompt. This means RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → applications with long retrieved contexts, or applications that stuff system prompts with many instructions, will have higher TTFT.
Infrastructure also affects TTFT: geographic distance to the inference server adds network round-trip time, queue depth adds waiting time, and whether the provider has GPU capacity to start immediately affects it significantly. Providers with dedicated capacity (vs. shared/spot) typically offer more consistent TTFT.
Model Size and Speed
Larger models are generally slower. A 70B parameter model generates fewer tokens per second than a 7B model on the same hardware, because each forward pass involves more computation. However, model architecture matters too: Mixture-of-Experts (MoE) models like Mixtral and Llama 4 Scout activate only a fraction of parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → per token, achieving large model quality at smaller model inference costs.
Specialized hardware changes the equation significantly. Groq's LPU architecture achieves extraordinary TPS — often 10-20x faster than GPU-based inference for the same model — by eliminating memory bandwidth bottlenecks. For applications where generation speed is critical (real-time voice, live coding assistants), hardware choice can matter as much as model selection.