GRPO — Group Relative Policy Optimization (DeepSeekMath, R1).
hardAnswer
- PPO variant that skips the value network.
- For each prompt, sample G responses, compute rewards, normalize as advantages: .
- Uses group-normalized rewards as advantage estimates → no critic needed.
- Preserves PPO's clipped surrogate loss.
- Enables efficient RL fine-tuning of reasoning models (DeepSeek-R1).
- Modern efficiency win for LLM RL.
Check yourself — multiple choice
- Same as PPO
- Sample G responses per prompt, normalize rewards as advantages ; no critic needed; foundation of DeepSeek-R1 reasoning RL
- Random
- Not real
GRPO: group-normalized rewards as advantages; no critic; DeepSeek-R1.
#llm#alignment#policy-methods
Practise Reinforcement Learning
214 interview questions in this topic.