Mode / representation collapse in SSL — what and why?
hardAnswer
- Encoder outputs the same (or trivially degenerate) representation for all inputs → useless embeddings.
- Root cause: shortcut solutions minimizing loss without capturing content.
- Prevention: (1) contrastive negatives, (2) asymmetry (stop-gradient + predictor, BYOL / SimSiam), (3) centering + sharpening (DINO), (4) variance-covariance regularization (VICReg, Barlow Twins), (5) predictor bottleneck.
- Understanding when/why collapse is avoided is an open research direction.
Check yourself — multiple choice
- Not real
- Encoder outputs trivial/degenerate representation; prevented by: contrastive negatives, stop-grad + predictor asymmetry (BYOL/SimSiam), centering (DINO), var-cov regularization (VICReg/Barlow)
- Random
- Only supervised
SSL collapse: prevented via contrast / stop-grad asymmetry / centering / var-cov.
#representation-learning#theory
Practise Unsupervised Learning
214 interview questions in this topic.