Greedy decoding vs sampling — when do you pick each?
easyAnswer
- Greedy (T=0): argmax at each step.
- Deterministic, reproducible, often suboptimal on open-ended tasks because it commits to the highest-probability token even when it leads to bad continuations.
- Sampling (T>0 + top-p/top-k): stochastic, produces more varied and often more natural outputs.
- Best practice: greedy or very low T for factual / code / structured; T=0.7-1.0 with for creative / conversational; T=0.2-0.5 with for balanced.
Check yourself — multiple choice
- Sampling is always worse
- Greedy/T=0: factual/code/structured; : creative; T=0.2-0.5: balanced
- Random sampling only
- No difference
Greedy: factual/code; sampling with T + top-p: creative/conversational.
#generation#decoding#sampling
Practise LLMs & GenAI
214 interview questions in this topic.