What is the Markov property and why does it matter?
mediumAnswer
- — the future depends only on the current state, not the history.
- Enables efficient RL: policy and value functions depend on state alone, not history.
- When violated (partial observability), we use POMDPs, recurrent nets, or state-stacking (Atari frame stack) to recover a Markovian representation.
Check yourself — multiple choice
- Random
- is independent of history; violated → POMDP + RNN or state-stacking
- Same as Bellman
- Not real
Markov: future depends only on (s, a); POMDP + RNN / stack when violated.
#fundamentals#theory
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- POMDP — how does it differ from MDP?
- How do you handle partial observability in deep RL?
- How do you choose the discount factor, and what goes wrong at the extremes?
- Define the RL setting: agent, environment, state, action, reward, policy.
- On-policy vs off-policy learning — what's the difference?
- What does the Bellman equation say?