Top-K.
A sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.
Top-k sampling is a decoding strategy that limits the model's choices at each step to the k tokens with the highest predicted probabilities. For example, with k=50, the model considers only the 50 most likely next tokens and samples from them according to their relative probabilities. Tokens outside the top-k are assigned zero probability.
Top-k was one of the earliest sampling strategies to improve over pure greedy decoding. It prevents the model from selecting highly improbable (and often nonsensical) tokens while still allowing creative variation. However, its fixed k can be problematic: on some steps the model is very confident and k=50 is too permissive, while on others it's spread out and k=50 is too restrictive.
This limitation motivated the development of top-p (nucleus) sampling, which adapts the number of candidates based on the actual probability distribution shape. In practice, many deployments use both: a top-k filter is applied first, then top-p, then temperature. This combination gives fine-grained control over the balance between coherence and diversity.
Related terms.
- SamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.
- Top-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.