What is a 'foundation model' in the unsupervised sense?
mediumAnswer
- Large model pretrained on massive unlabeled data via SSL → generalizes to many downstream tasks with little or no fine-tuning.
- Examples: GPT / Llama (text), CLIP / DINOv2 (vision), SAM (segmentation), Whisper (speech), ImageBind (multimodal).
- Bommasani et al. 2021 coined the term.
- Enables the modern 'pretrain once, adapt many' paradigm — economically transformative in industry.
Check yourself — multiple choice
- Random
- Large SSL-pretrained model generalizing across tasks with little/no fine-tuning; GPT / CLIP / DINOv2 / SAM / Whisper — 'pretrain once, adapt many' paradigm
- Same as classifier
- Not real
Foundation model: massive SSL pretrained; transfers broadly across tasks.
#representation-learning#applications
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Node2Vec — how does it learn graph node embeddings?
- Self-supervised graph learning — approaches.
- Self-supervised audio — wav2vec 2 and HuBERT.
- How are multimodal embeddings unified across text / image / audio?
- Fine-tuning vs linear probe vs prompt-tuning — when do you use each?
- What is self-supervised learning and why does it matter?