CLIP — how does contrastive image-text training work?

medium

Answer

$Train two encoders (image ViT + text Transformer) so that matching (image, caption) pairs are close in shared embedding space, mismatched pairs far.$
$InfoNCE loss over batch.$
$Trained on 400M image-caption pairs from web.$
$Result: zero-shot image classification via 'a photo of a {class}' prompt, cross-modal retrieval, foundation for DALL-E / Stable Diffusion.$
$Rewrote vision-language modeling; superseded by SigLIP, EVA-CLIP, SAM 2.$

Check yourself — multiple choice

Random
Image + text encoders → shared embedding; InfoNCE on 400M (image, caption) pairs; enables zero-shot classification + cross-modal retrieval; foundation of DALL-E / SD
Same as GPT
Not real

CLIP: image + text contrastive on 400M pairs; zero-shot / retrieval foundation.

#representation-learning

Practise Unsupervised Learning

214 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions