EasyDeepLearn

Policy iteration vs value iteration.

medium

Answer

  • Policy iteration: alternate (a) policy evaluation (compute  Vπ  by  solving  Bellman  until  convergence)(\mathrm{compute}\;V{\pi}\;\mathrm{by}\;\mathrm{solving}\;\mathrm{Bellman}\;\mathrm{until}\;\mathrm{convergence}), (b) policy improvement (π  ←  greedy  on  Vπ)({\pi}\; \leftarrow \;\mathrm{greedy}\;\mathrm{on}\;V{\pi}).
  • Converges in finite steps.
  • Value iteration: V ← max⁡a\operatorname{max}_{a} (r + γ V(s')) repeatedly — combines both in one step.
  • Value iteration is a special case of policy iteration with only one evaluation sweep.
  • Both are dynamic programming, require the model.
Check yourself — multiple choice
  • Same thing
  • PI: alternate eval-until-convergence + greedy improvement (finite steps); VI: V ← max⁡a\operatorname{max}_{a} (r + γ V(s')) one step (special case of PI); both are model-based DP
  • Random
  • Not real

PI: eval + improve alternation; VI: one-step Bellman iteration.

#theory

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions