The reward arrives only at the end of a 500-step episode. What do you do?
hardAnswer
- Shorten the effective credit assignment problem.
- Reward shaping is the direct approach, adding intermediate signal, and if you shape it as a potential-based difference the optimal policy provably does not change, which is the only shaping I would introduce without careful evaluation.
- Curriculum learning starts with short easy episodes and lengthens them, so the agent learns the final part of the task before the whole.
- Hindsight relabelling turns a failed trajectory into a successful one for whatever goal it actually reached, which manufactures dense signal from the same data in goal-conditioned settings.
- Better value estimation helps too, via generalized advantage estimation to trade bias against variance.
- And demonstrations, even a handful, bypass the search entirely for the early phase.
Check yourself — multiple choice
- Raise the learning rate
- Add potential-based shaping, build a curriculum of shorter episodes, use hindsight relabelling for goal-conditioned tasks, tune advantage estimation, and seed with demonstrations
- Reduce the discount factor to 0
- Sparse rewards cannot be handled
Potential-based shaping, curricula, hindsight relabelling and demonstrations all shorten the credit assignment path.
#reward-design#exploration
Practise Reinforcement Learning
214 interview questions in this topic.