Linear function approximation for value functions — pros and cons.
mediumAnswer
- V(s) ≈ w'φ(s) with hand-crafted features φ.
- Convergence guarantees under on-policy (LSPI, LSTD) but poor scaling; requires domain-designed features.
- Baird's counterexample: off-policy + linear FA + bootstrap → divergence.
- Foundation of classic RL theory (Sutton, Bertsekas), superseded by neural nets in practice.
- Still valuable in tiny state spaces, small-embedded systems, or theoretical proofs.
Check yourself — multiple choice
- Random
- V ≈ w'φ(s); on-policy convergence (LSPI/LSTD); off-policy + linear FA + bootstrap = Baird's divergence; classic theory foundation, superseded by NNs
- Same as NN
- Not real
Linear FA: on-policy converges; off-policy + FA + boot = Baird divergence.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.