How do you fix length bias in RLHF?
mediumAnswer
- (1) Length-normalize reward: divide by response length (log or square-root).
- (2) Add length penalty λ · length to policy loss.
- (3) In DPO: length-normalize log-probs.
- (4) Curate preference pairs to be length-matched.
- (5) Train RM to explicitly discount length (RM sees length as feature and is penalized for using it).
- Standard fix in modern LLM training; without it, models become verbose.
Check yourself — multiple choice
- Ignore
- Length-normalize reward (÷ length or √length) / add penalty λ·length / length-matched preference pairs / train RM to discount length; without fixes, models get verbose
- Random
- Not real
Length bias fix: normalize reward / add penalty / length-matched pairs / RM discount.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.