InfoNCE loss — formula and intuition.

hard

Answer

$[\operatorname{exp}(s(x, \;x^{+}) / {\tau})\; / \;{\Sigma}_{j}\;\operatorname{exp}(s(x, \;x_{j}) / {\tau})]$
$Cross-entropy that treats classification 'which of N is the positive?'.$
$x^{+}$
$Standard loss for contrastive SSL (SimCLR, CLIP, MoCo, etc.).$
$Temperature τ controls concentration: low τ \to sharp, hard-negative-focused.$

Check yourself — multiple choice

Random
-log $[\operatorname{exp}(s(x, x^{+}) / {\tau})\; / \;{\Sigma}_{j}\;\operatorname{exp}(s(x, x_{j}) / {\tau})]$ — softmax cross-entropy 'which is the positive'; lower bound on $\mathrm{MI}(x, \;x^{+})$ ; τ controls concentration
Same as MSE
Not real

InfoNCE: softmax over positive vs negatives; MI lower bound; τ sharpness.

#representation-learning#theory

214 interview questions in this topic.