Sampling.
The method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.
When an LLM generates text, it produces a probability distribution over its entire vocabulary for each next token. Sampling is the algorithm that picks one token from this distribution. The simplest approach is greedy decoding (always pick the highest-probability token), but this often produces repetitive, boring text.
More sophisticated sampling methods introduce controlled randomness. Temperature scaling flattens or sharpens the distribution. Top-p (nucleus) sampling restricts choices to the smallest set of tokens whose cumulative probability exceeds p. Top-k sampling limits choices to the k highest-probability tokens. These parameters work together to balance coherence and creativity.
The right sampling configuration depends on the task. Creative writing benefits from higher temperature and more permissive sampling. Code generation, mathematical reasoning, and factual Q&A benefit from lower temperature and more deterministic decoding. Most APIs expose temperature as the primary user-facing parameter, with defaults tuned for general helpfulness.