MoCo (Momentum Contrast) — how does it enable smaller batches?

hard

Answer

$Maintain a queue of negatives from previous batches instead of relying on current batch only.$
$Momentum encoder: EMA of the main encoder \to consistent keys in queue as encoder updates slowly.$
$Enables SimCLR-level performance with batches of 256 instead of 4096.$
$MoCo v2 / v3 improved augmentations, projection head, ViT backbone.$
$Foundation of large-scale efficient SSL.$

Check yourself — multiple choice

Random
Queue of past-batch negatives + momentum EMA encoder for consistent keys → SimCLR-level with small (256) batches
Same as SimCLR
Not real

MoCo: queue of negatives + momentum encoder; SSL with small batches.

#representation-learning

214 interview questions in this topic.