What do temperature and top-p do at generation time?

easy

Answer

$Temperature scales logits before softmax: low T (e.g., 0-0.3) makes outputs sharp and deterministic — use for factual answers and code.$
$High T (0.7-1.2) makes outputs more diverse — use for creative writing.$
$Top-p (nucleus) restricts sampling to the smallest set of tokens whose cumulative probability exceeds p.$
$\mathrm{top}_{p} = 0.9$

Check yourself — multiple choice

Higher temperature makes the model more deterministic
Top-p keeps the smallest set of tokens with cumulative $\mathrm{probability}\; \ge \;p$
Temperature and top-p do the same thing
Lower temperature increases hallucinations

Nucleus sampling caps the token pool by cumulative probability.

#generation#decoding#sampling

214 interview questions in this topic.