V(s) vs Q(s, a) — the difference.
easyAnswer
- V(s) = expected return from state s following policy π.
- Q(s, a) = expected return from state s taking action a THEN following π.
- Relationship: Q(s, a).
- Optimal: Q*(s, a).
- Q allows action selection without a model (Q-learning); V requires a model to compute argmax over actions.
- Actor-critic methods often estimate V (simpler) but use per-action advantage A = Q - V.
Check yourself — multiple choice
- Same thing
- V(s) = expected return from s under π; Q(s, a) = expected return doing a then π; V = Σ Q(s, a); Q enables argmax action selection without model
- Random
- V is Q
V(s) = state value; Q(s,a) = state-action value; V = Σ π Q.
#fundamentals#value-methods
Practise Reinforcement Learning
214 interview questions in this topic.
Related questions
- Define the RL setting: agent, environment, state, action, reward, policy.
- On-policy vs off-policy learning — what's the difference?
- What update does Q-learning perform?
- What role does the discount factor gamma play?
- What are the key tricks that make DQN work?
- Model-based vs model-free RL — when do you prefer each?