Why do Bellman operators converge?
hardAnswer
- Bellman operator T is a γ-contraction in the max norm: ||||_∞ ≤ γ ||||_∞.
- By Banach fixed-point theorem, iterating T converges to unique fixed point V* at rate .
- Underlies value iteration, TD(0) convergence in tabular case, policy iteration.
- Function approximation can break the contraction (projected Bellman operator may not contract) → divergence.
Check yourself — multiple choice
- Random
- T is a γ-contraction in max-norm → Banach fixed-point → unique V* + convergence rate; function approximation can break contraction (projected T may not contract)
- Not real
- Same as GD
Bellman: γ-contraction → Banach → convergence; FA can break it.
#theory
Practise Reinforcement Learning
214 interview questions in this topic.