EasyDeepLearn

What does the Bellman equation say?

medium

Answer

  • For a policy pi: Vπ(s)  =  E[r  +  γ  ⋅  Vπ(s)]V_{\pi}(s)\; = \;E[r\; + \;\gamma\; \cdot \;V_{\pi}(s)].
  • For the optimal value: V⋅(s)  =  max⁡aV \cdot (s)\; = \;\operatorname{max}_{a} E[r + gamma · V*(s')].
  • It expresses the value of a state as the immediate reward plus the discounted value of the next state.
  • Solving the Bellman equation is the core of dynamic programming, TD learning, and Q-learning.
Check yourself — multiple choice
  • V(s) = r + V(s') without discount
  • Vπ(s)  =  E[r  +  γ  ⋅  Vπ(s)]V_{\pi}(s)\; = \;E[r\; + \;\gamma\; \cdot \;V_{\pi}(s)]
  • The Bellman equation applies only to episodic tasks
  • It says V(s) = R for all s

Recursive value = expected immediate reward + discounted next value.

#theory

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions