REINFORCE — the vanilla policy-gradient algorithm.
mediumAnswer
- For episode τ with return G(τ), update θ ← θ + α G(τ) ∇_θ log π_θ summed over trajectory.
- Unbiased gradient of expected return via log-derivative trick.
- Very high variance: rewards accumulate over full episode → large gradient variance.
- Fixes: (1) subtract baseline b(s) (state-dep baseline / value function), (2) use advantage A(s, a) instead of raw G, (3) reward-to-go instead of full return.
Check yourself — multiple choice
- Random
- θ ← θ + α G(τ) ∇log π_θ summed over trajectory; unbiased but high variance → use baseline b(s), reward-to-go, or advantage A(s, a)
- Same as Q-learning
- Not real
REINFORCE: log-derivative trick with return G; reduce variance via baseline / advantage.
#policy-methods
Practise Reinforcement Learning
214 interview questions in this topic.