Learning-rate schedule for PPO / SAC — what works?
mediumAnswer
- PPO: linear anneal from 3e-4 to 0 over training.
- Adam optimizer, .
- SAC: fixed 3e-4 typically.
- LLM RLHF: much smaller LR (1e-6 to 1e-5) since starting from a well-trained model; sometimes with cosine schedule.
- Never use aggressive schedules that spike — RL is fragile to LR.
- Warmup helps prevent initial destabilization when starting fresh.
Check yourself — multiple choice
- Any high LR
- PPO: linear anneal 3e-4→0 with Adam; SAC: fixed 3e-4; RLHF: 1e-6 to 1e-5 with cosine; warmup helps fresh starts
- Random
- Not real
LR: PPO 3e-4 linear anneal; SAC fixed; RLHF 1e-6 to 1e-5.
#engineering
Practise Reinforcement Learning
214 interview questions in this topic.