- LLMs generate probability distributions over 30,000-150,000 tokens at each step, reflecting likelihood of each token following current context.
- Temperature adjusts randomness from 0 (deterministic) to 2.0+ (highly creative), with 0.7 being a common balanced default.
- Restricts sampling to K highest-probability tokens, with K=50 being a typical setting that ignores the rest of vocabulary.
- Dynamically includes tokens until cumulative probability reaches P (0.9-0.95), creating adaptive sampling sets based on model confidence.
- Most systems combine temperature with top-P filtering, while top-K is less commonly used in current practice.
The Probability Distribution Over Tokens
At each generation step, an LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → produces a probability distribution over its entire vocabularyVocabularyThe fixed set of tokens (words, subwords, characters) that a language model can recognize and generate, typically ranging from thousands to millions of unique elements.Learn more → — typically 30,000–150,000 possible tokens. The distribution reflects how likely each tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → is to follow the current context. 'Paris' might have 0.7 probability after 'The capital of France is', while 'Lyon' might have 0.05 and 'cheese' might have 0.001.
The samplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more → strategy determines how to convert this distribution into a concrete token choice. Different strategies make different tradeoffs between determinism (always picking the highest-probability token) and diversity (sometimes choosing lower-probability, more surprising tokens).
Temperature: The Master Dial
TemperatureTemperatureA parameter controlling the randomness of model outputs — lower values produce more focused, deterministic responses; higher values produce more creative, varied text.Learn more → is a scalar applied to the logitsLogitsRaw, unnormalized numerical scores that language models output for each possible next token before applying softmax to convert them into probabilities.Learn more → (raw scores) before computing probabilities. A temperature of 1.0 leaves the distribution unchanged. Lower temperatures (0.1–0.7) sharpen the distribution — the highest-probability tokens become even more likely, reducing randomness. Higher temperatures (1.2–2.0) flatten the distribution — all tokens become more equally likely, increasing creativity and diversity.
Temperature 0 (greedy decodingGreedy DecodingA text generation strategy that always selects the token with the highest probability at each step, producing deterministic but potentially repetitive output.Learn more →) always picks the single highest-probability token. This is fully deterministic and often produces high-quality but repetitive output. Temperature settings around 0.7 are a common default balancing quality and variety. Creative applications like story generation typically use 0.8–1.1; factual applications use 0.2–0.5.
Top-K and Top-P Filtering
Top-KTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → sampling restricts the sampling pool to the K highest-probability tokens, redistributing probability mass among only those tokens before sampling. Top-K = 1 is equivalent to greedy decoding. Top-K = 50 considers the 50 most likely options at each step, ignoring the rest of the vocabulary.
Top-PTop-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.Learn more → (nucleus sampling) is more dynamic: it includes tokens in order of descending probability until their cumulative probability reaches P (e.g., 0.9 or 0.95), then samples from that adaptive set. When the model is confident, the nucleus is small (few tokens); when uncertain, it's large. Most modern systems use temperature + top-P together, with top-K less common in practice. These parametersParametersThe numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.Learn more → work together: temperature shapes the distribution, top-P or top-K then filters it.