V-trace off-policy correction formula.

hard

Answer

$v_{s}\; = \;V(s_{s})\; + \;{\Sigma}_{t}$
$Different clipping thresholds for ρ (target) vs c (trace) allow bias-variance trade.$
$Off-policy corrections without exploding variance.$
$Enables IMPALA-style distributed training.$
$Modern reference for scalable off-policy actor-critic.$

Check yourself — multiple choice

Random
$v_{s}\; = \;V(s_{s})\; + \;{\Sigma}$ ${\gamma}^{t - s}$ $({\Pi}\;{\rho}_{i})$ ${\delta}_{t}^{V}$ with ${\rho}_{i}\; = \;\operatorname{min}({\rho}, \;{\pi} / {\mu})$ ; separate clipping thresholds for target and trace → tunes bias-variance; enables IMPALA
Same as GAE
Not real

V-trace: truncated IS + clipping ρ̄ for target and c for trace; IMPALA.

#actor-critic#theory

214 interview questions in this topic.