What does temperature do in a softmax?
easyAnswer
- : T > 1 flattens the distribution (softer, higher entropy — used in distillation for teacher outputs), T < 1 sharpens toward one-hot (used in decoding to make choices more deterministic), T = 0 is argmax.
- In LLM decoding, temperature is a knob for creativity vs determinism.
- In distillation, teacher and student use matching high T so students learn full distribution info.
Check yourself — multiple choice
- Temperature only affects training loss
- T > 1 softens the distribution; T < 1 sharpens; T = 0 is argmax
- Temperature is only used in convolutional nets
- T = 1 always gives one-hot outputs
Temperature scales logits — higher T = flatter distribution, lower T = sharper.
#fundamentals#activations#distillation
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Depth vs width — which do you scale first?
- Why can't you initialize a neural net with all zeros?