Verifier reward vs preference-based reward — when to use each?
mediumAnswer
- Verifier: objective correctness (math, code, structured tasks) — automatic, cheap, unambiguous, scalable.
- Best when correctness is well-defined.
- Preference: subjective quality (helpfulness, tone, style, creative writing) — needs human/AI labels, more ambiguous.
- Best when 'right' is unclear.
- Modern LLM training combines both: verifier RL for skills (o1 reasoning), preference RLHF for chat quality, safety.
Check yourself — multiple choice
- Same
- Verifier: objective correctness (math/code) — automatic + scalable + unambiguous; Preference: subjective quality (chat/style) — needs labels + ambiguous; modern LLMs combine both
- Random
- Only preference
Verifier vs preference: verifier for objective correctness, preference for subjective quality.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.