- Groq achieves 500-800 tokens per second on Llama 3.1 70B, five to ten times faster than GPU inference.
- Gemini 2.5 Flash delivers under 500ms TTFT and 100-plus TPS as the fastest large capable model.
- Streaming reduces perceived latency by three to five times even when total generation time is identical.
- Use Groq with Llama 3.3 70B or Gemini 2.5 Flash for lowest latency voice applications.
- Test latency from your deployment region as infrastructure geography significantly affects TTFT.
Why Speed Matters
For conversational AI, voice interfaces, and real-time coding assistants, latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → is not a nice-to-have. It is a core product requirement. Users start feeling friction when responses take more than one to two seconds to begin, and abandon workflows when generation is slow enough to disrupt their thinking.
Two metrics matter: Time to First TokenTime to First TokenTime to First Token (TTFT) measures the latency from sending a request to receiving the first streamed token, primarily determined by the prefill phase.Learn more → (TTFT), which measures how long until text starts appearing, and tokens per secondTokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.Learn more → (TPS), which measures how fast text streams after it begins. TTFT dominates perceived responsiveness. TPS determines how long users wait for long responses.
Speed Rankings
Groq leads on raw speed for smaller models, achieving 500 to 800 tokens per second on Llama 3.1 70B using custom LPU hardware. This is five to ten times faster than GPU inferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more →. For small models where speed is everything, Groq's API is unmatched. Cerebras offers similar performance on Llama variants.
Among frontier modelFrontier ModelThe most advanced and capable AI models at the cutting edge of the field, typically developed by well-funded research labs with massive computational resources.Learn more → providers, Gemini 2.5 Flash is the fastest large capable model, typically delivering under 500ms TTFT and 100-plus TPS. Claude 3.5 Haiku and GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o Mini are close competitors. Claude Opus 4 and GPT-5 are the slowest frontier models due to their size.
Streaming vs. Non-Streaming
All major providers support streamingStreamingStreaming returns a model's tokens incrementally as they're generated rather than waiting for the complete response to finish.Learn more → responses, which dramatically improves perceived responsiveness. With streaming, users see tokens as they are generated rather than waiting for the complete response. For conversational UIs, always enable streaming. It reduces perceived latency by three to five times even when total generation time is identical.
Reasoning models have inherently higher TTFT because they think before responding. Users see no output during the thinking phase, making them feel slower than standard models. Some providers expose thinking progress indicators to mitigate this.
Speed Recommendations
Voice AI: Use Groq with Llama 3.3 70B or Gemini 2.5 Flash for the lowest latency. Real-time coding autocomplete: GPT-4o Mini or Gemini 2.5 Flash Lite. Customer support chatbot: Claude 3.5 Haiku or GPT-4o Mini, both offering sub-second TTFT. Long-form generation where speed matters less: Claude Sonnet or GPT-4o are fine.
Test latency from your deployment region, as infrastructure geography significantly affects TTFT. A model that is fastest in the US may not be fastest for European or Asian users. Most providers offer regional endpoints or have global infrastructure.