How does knowledge distillation work?

medium

Answer

$Train a small 'student' network to match the outputs (usually soft probabilities at high temperature T) of a large pretrained 'teacher'.$
$\mathrm{Loss}\; = \;{\alpha}\; \cdot \;\mathrm{CE}(\mathrm{student}, \;\mathrm{hard}_{\mathrm{labels}})\; + \;(1 - {\alpha})\; \cdot \;T^{2}\; \cdot \;\operatorname{KL}(\operatorname{softmax}(\mathrm{student} / T)\; \mid \mid \;\operatorname{softmax}(\mathrm{teacher} / T))$
$The soft targets carry more information than hard labels (they encode inter-class similarities), so the student often achieves 90-95% of the teacher's accuracy at 5-10x smaller size.$
$Foundational for model compression, mobile deployment, MLM speedups (DistilBERT).$

Check yourself — multiple choice

Train two teachers
Small student matches teacher's soft probs at high T → carries inter-class info; used for compression (DistilBERT)
Same as fine-tuning
Only for CNNs

KD: student learns from teacher's soft probs at high temperature → strong compression.

#distillation

214 interview questions in this topic.