How does batch size / rollout length affect PPO?
mediumAnswer
- Larger rollout length → lower variance advantage estimates (more MC-like), but slower feedback.
- Batch size : must be big enough for a stable gradient estimate; typical 2048-16384.
- Too small → noisy gradients + unstable KL.
- Too big → wasted computation.
- Modern LLM RLHF uses very large batch sizes (millions of tokens per update).
Check yourself — multiple choice
- Random
- Longer rollout = lower advantage variance but slower feedback; batch (actors × rollout) = 2048-16384 typical for control; RLHF batches are millions of tokens
- Not important
- Same as SL
PPO batch: 2048-16384 for control; RLHF batches much larger.
#policy-methods#engineering
Practise Reinforcement Learning
214 interview questions in this topic.