Why is PPO used in RLHF instead of vanilla REINFORCE?

hard

Answer

$REINFORCE has huge gradient variance in the LLM regime (long token sequences, sparse rewards) and requires many samples to reduce it.$
$PPO uses a clipped surrogate objective that bounds how far the policy can move per update — much more stable.$
$It also uses generalized advantage estimation (GAE) to reduce variance further.$
$Trade-off: PPO adds hyperparameters (clip ratio, GAE lambda, value function).$
$Modern DPO / GRPO variants further simplify by removing the value function entirely.$

Check yourself — multiple choice

REINFORCE is more stable
PPO: clipped surrogate + GAE → bounded updates and lower variance vs REINFORCE — critical for token-level LLM training
PPO removes the reward
Same as DPO

PPO: clipped surrogate + GAE → stable RL updates on LLMs; REINFORCE is too high-variance.

#rlhf

214 interview questions in this topic.