Process reward vs outcome reward for reasoning.
hardAnswer
- Outcome reward: only final answer scored.
- Simple but sparse — credit assignment across long CoT is hard.
- Process reward (PRM, Lightman et al. 2023): score each reasoning step for correctness.
- Denser signal, better for long CoT.
- Training PRMs requires per-step labels (expensive).
- Modern practice: mix outcome + process; auto-generate process labels via LLM verifier or Monte Carlo (reason forward from step, check outcome distribution).
Check yourself — multiple choice
- Same
- Outcome: only final answer, sparse credit; Process: per-step correctness score, denser signal; PRM (Lightman) expensive labels; modern auto-label via LLM verifier or Monte Carlo
- Random
- Not real
Outcome vs process reward: outcome sparse; process dense but expensive labels.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.