When does the sample inefficiency of on-policy methods stop mattering?

medium

Answer

$When samples are cheap and stability is expensive, which is the case with a fast parallelizable simulator.$
$If you can run thousands of environment steps per second across many workers, PPO's need for fresh data on every update costs wall-clock time you have, and you get back a method that is markedly less brittle and has far fewer hyperparameters that can silently destroy a run.$
$Off-policy methods such as SAC reuse a replay buffer and can be an order of magnitude more sample efficient, which is what you need when each interaction involves a real robot, a real user, or a slow physics engine.$
$The decision therefore rests on the cost of an environment step relative to the cost of engineering time spent stabilizing training.$

Check yourself — multiple choice

Never, off-policy always wins
When environment steps are cheap and parallelizable, on-policy stability is worth the sample cost; off-policy replay matters when each interaction is expensive
When the action space is discrete
Only in tabular settings

The tradeoff is the cost of a sample against the cost of instability and tuning.

#policy-methods#engineering

214 interview questions in this topic.