Monte Carlo methods in RL — when to use?
mediumAnswer
- Estimate V_π(s) or Q_π(s, a) via sample-average returns from full episodes.
- Unbiased (no bootstrap error), but only usable in episodic tasks + high variance.
- Every-visit MC: average over every visit to s in an episode.
- First-visit MC: only first occurrence.
- Rare in modern deep RL (TD methods better for long / continuous tasks) but foundation of return-conditioned methods and offline RL (Decision Transformer).
Check yourself — multiple choice
- Random
- Sample-avg returns from full episodes → unbiased, no bootstrap error, but only episodic + high variance; every-visit / first-visit variants; foundation of return-conditioned methods
- Same as TD
- Not real
MC: sample returns; unbiased, high var, episodic; every/first-visit variants.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.