How does RL train chain-of-thought reasoning?
hardAnswer
- Reward on FINAL answer only (verifier: math correct? code passes tests?).
- Intermediate tokens get gradient via policy gradient on the final reward, credit-assigned across the response.
- Long CoT emerges because: (1) longer thinking gives higher chance of correct answer, (2) reward differential per token pushes toward exploration of reasoning traces.
- Combined with KL penalty to preserve language + prevent gibberish → emergent step-by-step reasoning.
Check yourself — multiple choice
- Random
- Reward only on final answer via verifier; credit-assigned across intermediate tokens via PG; long CoT emerges because longer thinking → higher accuracy; KL prevents gibberish
- Same as SFT
- Not real
CoT RL: verifier reward on final answer; credit assignment grows reasoning.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.