What update does Q-learning perform?
mediumAnswer
- Q(s, a) <- Q(s, a) + alpha · .
- This bootstraps toward the maximum future value at s', which is why it's off-policy: the target uses the greedy next action regardless of what the agent actually did.
- With function approximation (DQN), you use a replay buffer and a target network to stabilize learning.
Check yourself — multiple choice
- The Q-learning target uses r only
- The target is r + gamma · Q(s', a) where a is the next action taken
- The target is r + gamma · ' Q(s', a')
- Q-learning does not use gamma
Q-learning bootstraps with max over next actions — that's what makes it off-policy.
#value-methods
Practise Reinforcement Learning
214 interview questions in this topic.