EasyDeepLearn
Reinforcement Learning · section 6 of 12

Reward design & shaping

3 interview questions on reward design & shaping, each answered in full. Free to read, no account needed.

What is reward shaping and what are its pitfalls?

medium
  • Reward shaping adds intermediate rewards to guide the agent, e.g., dense proxies instead of only a sparse success signal.
  • Done right (potential-based shaping) it preserves the optimal policy.
  • Done wrong, it introduces bias and the agent finds shortcuts that maximize the shaped reward without solving the true task — the classic 'reward hacking' problem.
#reward-designPermalink & quiz →

Reward engineering — practical guidelines.

medium
  • (1) Start with sparse task reward (1 for success, 0 otherwise) — cleanest signal.
  • (2) If too sparse, add potential-based shaping (safe).
  • (3) Test agent for reward hacking: does it optimize what you meant?
  • (4) Include safety / constraint penalties.
  • (5) Normalize magnitudes across components.
  • (6) Log all reward components separately for debugging.
  • (7) Iterate.
  • Interview red flag: agent 'learns' but does something weird → reward is wrong.
#reward-design#engineeringPermalink & quiz →

How do you catch reward hacking before it reaches production?

hard
  • Watch for the signature: reward climbing while every measure you actually care about stagnates or degrades.
  • That means you must instrument metrics that are deliberately not part of the reward, since a reward you optimize can no longer serve as its own audit.
  • Watch trajectories, not just aggregates, because hacking usually shows up as a bizarre but highly repeated behaviour that a mean hides.
  • Hold out an evaluation environment with slightly different dynamics, since a hacked policy exploits specifics and generalizes badly.
  • Then constrain rather than only penalize: hard action limits and termination conditions are more reliable than a negative reward term the agent can trade away against the main objective.
#reward-design#safety#evaluationPermalink & quiz →

Practise Reinforcement Learning

All 12 sectionsFlashcardsQuiz