DINO — self-distillation with no labels.
hardAnswer
- Student network learns to match teacher (EMA of student) on different augmentations of same image.
- Uses ViT backbone → produces amazing self-attention maps (unsupervised object segmentation emerges!).
- Centering + sharpening on teacher output prevents mode collapse.
- DINOv2 (Meta 2023) scaled it to 1B parameters + 142M images → foundation vision encoder rivaling CLIP.
Check yourself — multiple choice
- Random
- Student matches EMA-teacher on augmented views; centering + sharpening prevents collapse; ViT backbone → emergent segmentation attention maps; DINOv2 at 1B params
- Same as SimCLR
- Not real
DINO: self-distillation from EMA teacher; emergent segmentation on ViT.
#representation-learning
Practise Unsupervised Learning
214 interview questions in this topic.