Potential-based reward shaping — why is it safe?
hardAnswer
- Ng, Harada, Russell 1999.
- Add r'(s, a, s') = r(s, a, s') + γΦ(s') - Φ(s).
- Any potential function Φ is safe: doesn't change optimal policy (only value baseline).
- Because telescopes → same argmax over policies.
- Enables safe reward engineering without changing task.
- Standard technique in robotics reward design.
Check yourself — multiple choice
- Random
- r' = r + γΦ(s') - Φ(s); telescopes → same optimal policy for any Φ (only value baseline changes); safe reward engineering without altering task — Ng-Harada-Russell 1999
- Same as shaping
- Not real
Potential shaping: γΦ(s') - Φ(s) telescopes; preserves optimal policy.
#reward-design#theory
Practise Reinforcement Learning
214 interview questions in this topic.