SARSA update — on-policy analog of Q-learning.
mediumAnswer
- Q(s, a) ← Q(s, a) + α [r + γ Q(s', a') - Q(s, a)] where a' is the ACTUAL action taken from s' (sampled from current π).
- On-policy: target uses the policy's next action, not the greedy one.
- More conservative than Q-learning: converges to Q for the policy including its exploration (safer near cliffs).
- Expected SARSA: use E_π[Q(s', a')] instead of sampled a' → less variance.
Check yourself — multiple choice
- Same as Q-learning
- Q ← Q + α[r + γ Q(s', a') - Q(s, a)] with a' sampled from current π (on-policy); more conservative than Q-learning near cliffs; Expected SARSA uses E_π[Q]
- Random
- Not real
SARSA: on-policy, uses actual next action; conservative near cliffs.
#value-methods
Practise Reinforcement Learning
214 interview questions in this topic.