Two RL algorithms report different scores in a paper. Why should you be skeptical?
mediumAnswer
- Because reinforcement learning results have unusually high variance across seeds, and a difference computed from three runs is frequently noise.
- The distribution of final performance is often bimodal, with some seeds failing entirely, so a mean hides more than it shows and a maximum over seeds is not a result at all.
- Implementation details matter as much as the algorithm: observation normalization, advantage normalization, reward scaling and network initialization can move scores more than the contribution being claimed.
- Evaluation protocol also differs, in the number of episodes, whether the policy is stochastic or greedy, and whether scores come from training or separate rollouts.
- Ask for many seeds, confidence intervals or performance profiles, and identical tuning budgets for both methods.
Check yourself — multiple choice
- Papers are always reproducible
- Seed variance is large and often bimodal, implementation details can dominate the algorithmic contribution, and evaluation protocols differ — demand many seeds, intervals and equal tuning budgets
- Only the network size matters
- Variance is not an issue in RL
High seed variance and implementation sensitivity mean few-seed comparisons are usually uninformative.
#evaluation#engineering
Practise Reinforcement Learning
214 interview questions in this topic.