What is an actor-critic algorithm?
mediumAnswer
- Actor-critic combines a policy (actor) that picks actions and a value function (critic) that estimates the advantage or value.
- The critic reduces the variance of the policy-gradient estimate.
- Popular variants: A2C/A3C, PPO (clipped surrogate objective, on-policy), and SAC (off-policy, entropy-regularized).
- PPO is the workhorse for RLHF of language models.
Check yourself — multiple choice
- The actor estimates the value function
- The critic reduces variance of the policy-gradient estimate
- Actor-critic has only one network
- SAC is on-policy
Critic estimates V or Q so we can use advantages instead of raw returns.
#policy-methods#actor-critic
Practise Reinforcement Learning
214 interview questions in this topic.