How are multimodal embeddings unified across text / image / audio?
hardAnswer
- Approaches: (1) CLIP-style paired training (image ↔ text).
- (2) ImageBind (Meta 2023): 6 modalities aligned through image as the 'bridge' — no need for all-pair data.
- (3) Whisper + CLIP for text-audio alignment.
- (4) LLaVA-style: project image encoder into LLM embedding space.
- Common thread: contrastive alignment + a bridge modality.
- Foundation of multimodal LLMs.
Check yourself — multiple choice
- Same as CLIP only
- CLIP-paired (image-text) / ImageBind (image as bridge for 6 modalities) / LLaVA (project image into LLM embedding); contrastive + bridge modality
- Random
- Not real
Multi-modal embeddings: CLIP / ImageBind / LLaVA; contrastive + bridge.
#representation-learning#applications
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Node2Vec — how does it learn graph node embeddings?
- Self-supervised graph learning — approaches.
- Self-supervised audio — wav2vec 2 and HuBERT.
- Fine-tuning vs linear probe vs prompt-tuning — when do you use each?
- What is a 'foundation model' in the unsupervised sense?
- What is self-supervised learning and why does it matter?