What is deep supervision / auxiliary loss?

medium

Answer

$Attach small classification / regression heads to intermediate layers and add their losses to the main loss (with lower weights).$
$Provides gradient signal deeper into the network — helps optimization of very deep nets and enables intermediate feature usefulness.$
$Used in Inception (auxiliary classifiers), U-Net (deep supervision on multi-scale masks), and dense prediction.$
$Modern residual nets need it less.$

Check yourself — multiple choice

One loss only
Add small heads at intermediate layers, weight their losses → deeper gradient signal, useful mid-features
Removes main loss
Only for RNNs

Deep supervision: intermediate heads and losses → better mid-layer features, deeper gradient.

#losses#training

214 interview questions in this topic.