EasyDeepLearn

Write info-NCE / NT-Xent and its role in SSL.

hard

Answer

  • L(x,  x+,  x−)L(x, \;x^{+}, \;x^{-}) = -log⁡[exp⁡(sim(x,  x+)  /  τ)  /  (exp⁡(sim(x,  x+)  /  τ)  +  Σ  exp⁡(sim(x,  x−)  /  τ))]\operatorname{log}[\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; / \;(\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; + \;{\Sigma}\;\operatorname{exp}(\mathrm{sim}(x, \;x^{-})\; / \;{\tau}))], sim = cosine similarity, τ = temperature.
  • Treats contrastive learning as a classification problem: 'which of these candidates is the positive?'.
  • Foundation of SimCLR, MoCo, CLIP, GPT-style retrieval.
  • Larger negative batches → richer learning signal.
Check yourself — multiple choice
  • Same as triplet
  • InfoNCE: softmax over positive vs many negatives with cosine sim / temperature — SimCLR/MoCo/CLIP foundation
  • Removes negatives
  • Only for images

InfoNCE = softmax-classification over one positive vs many negatives; core to modern SSL.

#losses#contrastive#self-supervised

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions