On-policy vs off-policy learning — what's the difference?
mediumAnswer
- On-policy methods learn from data generated by the current policy (e.g., SARSA, REINFORCE, PPO).
- Off-policy methods learn from data collected by a different (behavior) policy — this enables replay buffers and reusing old data (e.g., Q-learning, DQN, SAC, DDPG).
- Off-policy is more sample-efficient; on-policy is often more stable and easier to tune.
Check yourself — multiple choice
- PPO is an off-policy algorithm
- DQN is on-policy
- Q-learning is off-policy; SARSA is on-policy
- On-policy allows replay buffers
Q-learning bootstraps with ' Q(s',a') — off-policy. SARSA uses the actual next action — on-policy.
#fundamentals
Practise Reinforcement Learning
214 interview questions in this topic.