How are 'reasoning models' like o1 / R1 trained?
hardAnswer
- Combine SFT on long chain-of-thought traces (often self-generated with rejection sampling on math / code datasets) with RL that rewards correct final answers on verifiable tasks (math with numerical checkers, code with unit tests).
- The model learns to spend variable inference-time compute — long internal 'thinking' before emitting the final answer.
- DeepSeek-R1 revealed the recipe publicly: GRPO on verifiable rewards + curriculum from easier to harder problems.
Check yourself — multiple choice
- Random chain-of-thought
- SFT on long CoT + RL with verifiable rewards (math/code) → variable inference-time thinking (o1, DeepSeek-R1)
- No RL involved
- Only classification
Reasoning models: SFT on long CoT + RL with verifiable rewards → inference-time thinking budget.
#alignment#fine-tuning
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?
- How is high-quality instruction-tuning data constructed?
- Fine-tuning vs RAG — which do you use when?
- What is LoRA and why is it popular for fine-tuning LLMs?
- What is continual pretraining and when does it beat fine-tuning?
- Describe the full RLHF pipeline in three stages.