How does CLIP enable multimodal capability?
mediumAnswer
- CLIP (Radford 2021): trained a text encoder and image encoder jointly with contrastive loss on 400M web (image, caption) pairs.
- Aligns them in a shared embedding space — same-semantic image and caption have similar embeddings.
- Enables: zero-shot image classification (embed candidate labels, pick nearest), image search from text, cross-modal retrieval.
- Foundation for later multimodal LLMs — most VLMs use a CLIP-style encoder to embed images before passing to the transformer.
Check yourself — multiple choice
- Text only
- Contrastive training of joint image + text encoders on 400M pairs → shared embedding space; foundation of modern VLMs
- Same as ResNet
- Only for classification
CLIP: contrastive joint image-text encoders → shared embedding → foundation for VLMs.
#multimodal#embeddings
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How does multimodal retrieval (image + text) work?
- What are text embeddings and how are they used?
- How do you pick an embedding model for a new RAG project?
- What's the trade-off between embedding dimension and retrieval quality?
- What is Matryoshka Representation Learning (MRL)?
- How is ColBERT different from a standard bi-encoder?