LLM Inference: How Models Generate Text

Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.

Key takeaways
  • LLM inference generates text autoregressively, producing one token at a time with each token conditioned on all previous tokens.
  • Processes the entire input prompt in parallel to compute key-value representations, determining time-to-first-token latency.
  • Generates output tokens sequentially through full forward passes, with tokens-per-second measuring decode throughput.
  • 70B parameter models require roughly 140 GB of GPU memory just for weights, necessitating multiple high-end GPUs.
  • Processing multiple requests together amortizes memory bandwidth costs and dramatically improves throughput while reducing per-request cost.

What Is LLM Inference?

InferenceInferenceThe process of running a trained AI model to generate outputs — what happens when you send a prompt and receive a response.Learn more → is the process of running a trained model to generate outputs. When you send a promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more → to an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → and receive a response, every step between your request and the first tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → of output is inference. It's distinct from training, which is the expensive process of updating model weights on large datasets — inference uses a fixed, trained model to produce predictions.

For text generation, inference is autoregressiveAutoregressiveA text generation approach where models produce sequences one token at a time, with each new token conditioned on all previously generated tokens.Learn more →: the model generates one token at a time, each conditioned on all previous tokens (the prompt plus already-generated text). This sequential dependency is the fundamental constraint that makes LLM inference harder to parallelize than training.

Prefill and Decode Phases

LLM inference has two distinct phases. PrefillPrefillThe initial inference phase where an LLM processes the entire input prompt in parallel to populate the KV cache before beginning sequential token generation.Learn more → processes the entire input prompt in parallel, computing the key-value (KV) representations for every input token at once. This is compute-bound and relatively fast per token. Decode generates output tokens one at a time, each step requiring a full forward pass through the model while attending to all previous tokens via the KV cacheKV CacheA memory optimization that stores previously computed attention keys and values so the model doesn't recompute them when generating each new token.Learn more →.

Time-to-first-token (TTFT) is largely determined by the prefill phase — longer prompts mean slower first tokens. After the first token, tokens-per-second (TPS) measures decode throughputThroughputThe number of tokens or requests an AI system can process per unit time, measuring overall capacity rather than per-request speed.Learn more →. LatencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more →-sensitive applications care most about TTFT; throughput-sensitive use cases (document processing, batch jobs) care more about TPS.

Hardware and Batching

LLM inference runs on GPUs (or TPUs) because matrix multiplications — the core operation — map naturally to parallel hardware. A 70B parameter model requires roughly 140 GB of GPU memory just to store weights in 16-bit precision, requiring multiple high-end GPUs. Inference providers like Together AI, Fireworks, and Groq have built specialized infrastructure to serve large models efficiently at scale.

Batching multiple requests together amortizes the memory bandwidth cost of loading model weights across many users, dramatically improving throughput and reducing per-request cost. Continuous batchingContinuous BatchingA serving optimization technique that dynamically adds and removes requests from GPU batches during execution to maximize throughput and hardware utilization.Learn more → (serving requests at different decode stages simultaneously) is a key technique that frontier inference providers use to maximize GPU utilization.

Read next
7 min read

Tokens and Tokenization: The Building Blocks of LLMs

Everything you need to know about tokens — how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.

5 min read

LLM Latency: What Makes Models Feel Fast or Slow

Two key metrics — time to first token and tokens per second — determine how responsive an LLM feels. Here's what drives each.

7 min read

Training vs Inference: Two Phases of an LLM's Life

Understand the difference between training an LLM (creating it) and inference (using it), including what happens at each stage, the costs involved, and why they matter for builders.

5 min read

KV Cache: How LLMs Remember Context Efficiently

The key-value cache is the mechanism that lets LLMs process long conversations without recomputing everything from scratch on every token.