TD vs MC — how they differ in bias-variance.
mediumAnswer
- MC: use full episodic return → unbiased estimator of V_π, but high variance (many random rewards over full trajectory).
- TD(0): use V(s') → biased (bootstrap error from imperfect V estimate) but lower variance (only one reward). n-step and TD(λ) interpolate.
- Choice: TD when episodes are long or variance dominates, MC when bias is intolerable + episodes short.
Check yourself — multiple choice
- Random
- MC: unbiased + high variance (full episode); TD(0): biased + low variance (one-step bootstrap); n-step or TD(λ) interpolate
- Same thing
- Only MC
MC unbiased high-var; TD(0) biased low-var; n-step / λ interpolate.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.