- Controls probability distribution concentration, with T < 1 making outputs more deterministic and T > 1 increasing randomness.
- Selects from the smallest token set reaching cumulative probability p, with 0.9-0.95 being common defaults.
- Restricts sampling to k most probable tokens, though most modern APIs prefer top-p for its adaptive behavior.
- Raw model scores are converted to probabilities via softmax function, maintaining relative token ordering.
- Temperature, top-p, and top-k work together in a three-stage pipeline for fine-grained sampling control.
- T = 0 for deterministic outputs, 0.2-0.5 for factual tasks, 0.7-1.0 for general use, and 1.0-1.5 for creative tasks.
From Logits to Probability Distributions
After a forward pass through the transformerTransformerThe neural network architecture that underpins virtually all modern LLMs, introduced in 2017, built around self-attention mechanisms that process entire sequences in parallel.Learn more → layers, the model produces logitsLogitsRaw, unnormalized numerical scores that language models output for each possible next token before applying softmax to convert them into probabilities.Learn more → — one raw score per vocabularyVocabularyThe fixed set of tokens (words, subwords, characters) that a language model can recognize and generate, typically ranging from thousands to millions of unique elements.Learn more → tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more →. These raw scores are converted to probabilities using the softmaxSoftmaxA mathematical function that converts a vector of raw logits into a probability distribution, with temperature controlling the sharpness of the output.Learn more → function: exponential of each logit divided by the sum of all exponentials. The resulting probabilities sum to 1 and represent the model's 'belief' about which token should come next.
A key property: the relative ordering of tokens doesn't change — the token with the highest logit always has the highest probability. What samplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more → controls is how concentrated versus spread out these probabilities are when choosing which token to actually select.
Temperature: The Creativity Dial
TemperatureTemperatureA parameter controlling the randomness of model outputs — lower values produce more focused, deterministic responses; higher values produce more creative, varied text.Learn more → T is applied before softmax: logits are divided by T. Low temperature (T < 1): dividing logits by a small number amplifies differences, making high-probability tokens much more likely. At T 0, the model always picks the highest-probability token (greedy decodingGreedy DecodingA text generation strategy that always selects the token with the highest probability at each step, producing deterministic but potentially repetitive output.Learn more →). High temperature (T > 1): dividing by a large number compresses differences, flattening the distribution toward uniform — all tokens become roughly equally likely.
Practical guidance: T = 0 for completely deterministic outputs (useful for testing and structured data extraction). T = 0.2–0.5 for factual, precise tasks (code generation, technical Q&A). T = 0.7–1.0 for general use (the default range). T = 1.0–1.5 for creative tasks (poetry, fiction, brainstorming). T > 1.5 produces increasingly incoherent output.
Top-P (Nucleus Sampling)
Top-pTop-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.Learn more → sampling selects from the smallest set of tokens whose cumulative probability reaches p. With p = 0.9, the model considers only the most probable tokens totaling 90% of the probability mass. On confident steps, this might be 2–3 tokens; on uncertain steps, hundreds. This adaptive behavior is its advantage over the fixed top-kTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → cutoff.
Top-p = 0.9–0.95 is the most common default setting across AI APIs. Lower values (0.7–0.8) produce more focused but potentially repetitive text. Values near 1.0 are approximately equivalent to no top-p restriction. Top-p and temperature interact: high temperature with high top-p is maximally random; low temperature with low top-p is nearly deterministic.
Top-K Sampling
Top-k restricts sampling to the k most probable tokens. k=1 is greedy; k=50 is a common default; k=∞ is equivalent to no restriction. Top-k is simpler than top-p but less adaptive — the same k value may be too restrictive on confident steps and too permissive on uncertain ones.
Most modern LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → APIs use top-p rather than top-k, or combine them: apply top-k first to remove the long tail of low-probability tokens, then apply top-p within the remaining candidates. Temperature is applied last. This three-stage pipeline gives fine-grained control over the entire sampling distribution.