- Dynamically selects the smallest set of tokens whose cumulative probability exceeds threshold P, adapting to model confidence.
- Common values range from 0.85 to 0.99, with 0.9 as conservative default and 0.95-0.99 more common in practice.
- Top-P is almost always used with temperature, where temperature reshapes probability distribution first, then top-P filters the result.
- When model is highly confident, nucleus contains few tokens; when uncertain, nucleus may contain hundreds of tokens.
- Temperature 0.7 with top-P 0.95 produces quality output with controlled diversity for most applications.
The Nucleus Sampling Intuition
Nucleus samplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more →, introduced in the 2020 paper 'The Curious Case of Neural Text Degeneration,' addresses a core limitation of top-KTop-KA sampling parameter that restricts token selection to the k most probable next tokens, discarding the long tail of unlikely options.Learn more → sampling: the optimal pool size depends on how concentrated the probability distribution is at a given step. The paper proposed a dynamic alternative: sample from the 'nucleus' — the smallest set of tokens whose cumulative probability exceeds a threshold P.
When P = 0.9, the model adds tokens in descending probability order until their cumulative probability reaches 90%, then samples from that set. If the model is highly confident (one tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → has 95% probability), the nucleus might contain just one or two tokens. If the model is very uncertain, the nucleus might contain hundreds. This adaptive behavior makes top-PTop-PAlso called nucleus sampling — a method that selects from the smallest set of tokens whose combined probability exceeds a threshold p, adapting dynamically to the model's confidence at each step.Learn more → more principled than top-K in varied text generation scenarios.
Choosing the Right P Value
Common top-P values range from 0.85 to 0.99. A value of 0.9 is a conservative default that keeps generation coherent while allowing modest creativity. Values of 0.95–0.99 are more common in practice, giving the model more freedom while still excluding the long tail of implausible tokens. Very low values (0.5–0.7) produce more conservative, predictable outputs similar to low temperatures.
Top-P is almost always used in combination with temperatureTemperatureA parameter controlling the randomness of model outputs — lower values produce more focused, deterministic responses; higher values produce more creative, varied text.Learn more →. Temperature reshapes the probability distribution first; then top-P filters the result. Setting both temperature and top-P: a common pattern is temperature = 0.7, top-P = 0.95, which produces quality output with controlled but real diversity. For deterministic or near-deterministic outputs (code, structured data), use temperature = 0 and ignore top-P, as greedy decodingGreedy DecodingA text generation strategy that always selects the token with the highest probability at each step, producing deterministic but potentially repetitive output.Learn more → bypasses sampling altogether.