Options framework — formalization.
hardAnswer
- Sutton, Precup, Singh 1999.
- An option = (I, π, β): initiation set I (states where the option can start), intra-option policy π, termination condition β(s) (probability of stopping).
- High-level policy chooses among options.
- Bellman equations extend naturally.
- Foundational abstraction for hierarchical RL.
- Modern: skill discovery methods (DIAYN, OPAL) auto-discover options from unsupervised interaction.
Check yourself — multiple choice
- Random
- Option = (I initiation set, π intra-option policy, β termination); high-level policy chooses among options; Bellman extends; foundation of HRL; DIAYN / OPAL auto-discover
- Same as MDP
- Not real
Options: (I, π, β); high-level chooses; foundation of HRL.
#policy-methods#theory
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- Why is subtracting a baseline OK in policy gradient?
- The log-derivative trick — why is it central to policy gradients?
- What is natural gradient in policy gradient methods?
- Reparameterization trick in stochastic policies.
- Off-policy policy gradient — how does importance sampling enter?
- Policy gradient as mirror descent — the connection.