- Top-K sampling considers only the K tokens with highest probability at each generation step.
- K = 1 produces deterministic output while K = 50 is a common default for balanced diversity.
- Top-K uses a fixed number regardless of model confidence, unlike adaptive top-P sampling.
- Top-P sampling is more common in deployments, though top-K remains in Google's Vertex AI models.
- Higher K values increase output diversity while lower K values maintain focus and predictability.
What Is Top-K Sampling?
Top-KTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → samplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more → is a decoding strategy that, at each tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → generation step, considers only the K tokens with the highest probability. The remaining vocabularyVocabularyThe fixed set of tokens (words, subwords, characters) that a language model can recognize and generate, typically ranging from thousands to millions of unique elements.Learn more → tokens — no matter how many — are discarded, and probability mass is redistributed among only the top-K candidates before sampling.
Setting K = 1 produces greedy decodingGreedy DecodingA text generation strategy that always selects the token with the highest probability at each step, producing deterministic but potentially repetitive output.Learn more → — the most likely token is always chosen, resulting in fully deterministic output. K = 50 is a common default in many systems. Higher K values allow the model to occasionally choose less obvious tokens, increasing output diversity; lower K values keep generation more focused and predictable.
Top-K vs. Top-P: Key Differences
The main limitation of top-K is that K is a fixed number regardless of the model's confidence. When the model is very confident (e.g., generating code with a clear next step), top-K = 50 may include many implausible tokens. When the model is uncertain (e.g., generating a creative story), top-K = 50 may exclude many reasonable continuations.
Top-PTop-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.Learn more → (nucleus sampling) adapts to confidence: it dynamically selects however many tokens are needed to cover P% of the probability mass. This makes top-P more situationally appropriate in most cases. Modern language model deployments commonly use top-P alone, top-P + temperatureTemperatureA parameter controlling the randomness of model outputs — lower values produce more focused, deterministic responses; higher values produce more creative, varied text.Learn more →, or no filtering (relying on temperature alone). Top-K is still used in some systems (notably Google's Vertex AI models) and some research applications.