What is reward shaping and what are its pitfalls?
mediumAnswer
- Reward shaping adds intermediate rewards to guide the agent, e.g., dense proxies instead of only a sparse success signal.
- Done right (potential-based shaping) it preserves the optimal policy.
- Done wrong, it introduces bias and the agent finds shortcuts that maximize the shaped reward without solving the true task — the classic 'reward hacking' problem.
Check yourself — multiple choice
- Reward shaping never changes the optimal policy
- Reward hacking is when the agent exploits shaped rewards without solving the task
- Sparse rewards are always easier to learn from
- Potential-based shaping is unsafe
Reward hacking: the agent optimizes what you measure, not what you meant.
#reward-design
Practise Reinforcement Learning
214 interview questions in this topic.