What is TD error?

medium

Answer

${\delta}_{t}\; = \;r_{t}\; + \;{\gamma}$
$Drives all TD-based updates: V \leftarrow V + α δ, Q-learning target is δ with max in bootstrap.$
$Positive δ = state was better than expected; negative = worse.$
$Fundamental signal for value learning.$
$Modern uses: PPO advantage, GAE, prioritized replay weighting (δ magnitude).$

Check yourself — multiple choice

Same as MSE
${\delta}_{t}\; = \;r_{t}\; + \;{\gamma}$ $V(s_{t + 1})\; - \;V(s_{t})$ — bootstrapped target minus current estimate; drives all TD updates + prioritized replay
Random
Not real

TD error δ = target - current; core signal for TD-based methods.

#theory#value-methods

214 interview questions in this topic.