BCQ — Batch-Constrained Q-Learning.
hardAnswer
- Fujimoto et al. 2019.
- Constrain policy to only take actions similar to those in the data.
- Uses a conditional VAE trained on (s, a) pairs to sample actions close to behavior.
- Q-learning then only over these sampled 'safe' actions.
- Effectively KL-constrained policy improvement without explicit KL.
- Modern successors: TD3+BC (adds BC term), IQL (uses expectiles).
Check yourself — multiple choice
- Same as SAC
- Constrain policy to actions similar to data; conditional VAE samples 'safe' actions; Q-learning restricted to these; foundation of TD3+BC / IQL modern offline RL
- Random
- Not real
BCQ: policy constrained to in-distribution actions via cVAE; foundation.
#offline-rl
Practise Reinforcement Learning
214 interview questions in this topic.