Scaling laws for self-supervised pretraining.
hardAnswer
- Downstream performance improves as a power law in (data, params, compute).
- MAE / CLIP / DINOv2 / vision transformers all show emergent capabilities at scale (rare classes, zero-shot).
- Diminishing returns: 10× more compute typically gives ~linear gain in loss.
- Data quality matters more than quantity at very large scale (DINOv2 uses curated 142M images vs LAION's 5B).
- Modern trend: careful data selection + scale.
Check yourself — multiple choice
- Random
- Power law in (data, params, compute); emergent capabilities at scale; data quality > quantity at extreme scale — DINOv2's 142M curated beats LAION's 5B
- Same as CNN
- Not real
SSL scaling: power law + emergence; data quality > quantity at extreme scale.
#representation-learning#theory
Practise Unsupervised Learning
214 interview questions in this topic.