Why does DQN scale poorly to continuous action spaces?
hardAnswer
- Q-learning requires Q(s, a) — over a continuous action space, this is a nested optimization at every step (expensive) or requires discretizing (loses fidelity).
- Solutions: (1) Actor-critic (DDPG, SAC): explicit π_θ(s) network outputs continuous action; (2) NAF (Normalized Advantage Function): analytic argmax via quadratic-in-a parameterization; (3) sample-based action selection with CEM (QT-Opt).
- Modern default: SAC + reparameterized Gaussian policy.
Check yourself — multiple choice
- Random
- Q(s, a) is expensive over continuous a → use actor-critic (DDPG/SAC) with explicit π_θ(s) or NAF's quadratic-in-a or CEM sampling; SAC default
- Same as DQN
- Not real
Continuous actions: Q hard → actor-critic or NAF or CEM; SAC default.
#value-methods#deep-rl
Practise Reinforcement Learning
214 interview questions in this topic.