Inference.
The process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.
Inference is the production-time operation of an AI model: taking an input (your prompt) and generating an output (the response). It is distinct from training, which is the process of updating model weights on large datasets. Once a model is trained, inference is run billions of times to serve users.
Inference is computationally expensive. A large model like GPT-4 requires dozens of powerful GPUs to run efficiently at scale. Inference costs directly determine API pricing, which is why model providers invest heavily in inference optimization. Techniques like quantization, speculative decoding, and batching reduce inference costs dramatically.
Two key inference metrics matter for users: latency (time to first token, TTFT) and throughput (tokens per second). Fast inference enables real-time applications like voice assistants and code completion. Providers like Groq have built specialized inference hardware achieving thousands of tokens per second on smaller models.
Related terms.
- Tokens per SecondTokens per second measures how quickly an LLM generates text during inference, serving as the primary throughput metric for comparing model speed.
- Time to First TokenTime to First Token (TTFT) measures the latency from sending a request to receiving the first streamed token, primarily determined by the prefill phase.