What is GRPO?
hardAnswer
- Group Relative Policy Optimization (DeepSeek 2024).
- Skips the value function of PPO.
- For each prompt, sample K completions, compute their rewards, and use their z-scored advantage relative to the group mean as the PPO gradient signal.
- Advantages: no value network to train (halves memory), simpler, works well for math/code where rewards are ± and rankings within a group are meaningful.
- Used to train DeepSeek-Math and DeepSeek-R1 (reasoning-focused models).
Check yourself — multiple choice
- GRPO uses a value network
- Group Relative Policy Optimization: sample K completions/prompt, z-score rewards → PPO without value network (DeepSeek-R1)
- Same as REINFORCE
- Only classification
GRPO: PPO without value network; z-scored group rewards → simpler and memory-efficient.
#rlhf#alignment
Practise LLMs & GenAI
214 interview questions in this topic.