CQL — Conservative Q-Learning.

hard

Answer

$Kumar et al. 2020.$
$Add penalty to standard Q-learning loss: expectation of Q over the policy (pushed down) minus expectation over data (pushed up) \to pessimistic on OOD actions.$
$Prevents Q from over-estimating outside data.$
$Widely used in modern offline RL benchmarks.$
$Result: policy converges to a lower-bound on true Q \to conservative, safe behavior.$
$Simple to implement on top of SAC.$

Check yourself — multiple choice

Same as SAC
Standard Q loss + penalty: E_π $[Q]\; - \;E_{\mathrm{data}}[Q]$ pushed down on OOD actions → pessimistic Q lower-bound; conservative safe offline policy; simple SAC add-on
Random
Not real

CQL: pessimism penalty pushes Q down on OOD actions; safe offline RL.

#offline-rl

214 interview questions in this topic.