Why is length normalization needed in beam search?
mediumAnswer
- Log-probability of a sequence is the sum of log — always negative, and longer sequences pile up more negatives.
- Without normalization, beam search prefers shorter sequences by construction.
- Length normalization divides by length^α (α ~ 0.6-1.0) or uses a length penalty.
- Standard in NMT and summarization.
- Modern LLM sampling uses temperature/top-p/top-k instead of beam search, so length norm is less critical there.
Check yourself — multiple choice
- Length norm shortens outputs
- Divide log-prob by length^α to prevent bias toward short sequences in beam search
- Only for CNNs
- Adds noise to logits
Length norm: prevents beam search from always picking the shortest hypothesis.
#transformers#rnn
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why did transformers replace RNNs for sequence modeling?
- What is teacher forcing and its main pitfall?
- How does beam search work and what is its main failure mode?
- How does self-attention work in a transformer?
- Why does a transformer need positional encoding?
- When is orthogonal weight initialization useful?