What is consistency regularization?
mediumAnswer
- Enforce that two augmented views of the same unlabeled input produce similar predictions: (f(aug1(x)), f(aug2(x))) with D = KL or MSE.
- Powers modern semi-supervised methods (FixMatch, Mean Teacher, MixMatch) and self-supervised learning (SimCLR, MoCo, BYOL).
- Cornerstone technique when labeled data is scarce.
Check yourself — multiple choice
- Consistency means identical labels
- Two augmentations of the same input should predict similarly — cornerstone of SSL / semi-supervised
- Only for supervised learning
- Adds noise to labels
Consistency: two views ⇒ similar predictions — key to FixMatch / SimCLR / BYOL.
#regularization#self-supervised#augmentation
Practise Deep Learning
214 interview questions in this topic.
Related questions
- How does Mixup work?
- What is Cutout / Random Erasing?
- Why is data augmentation effectively free regularization?
- Your validation loss is lower than your training loss. Is something broken?
- How does dropout work and when is it applied?
- Why does AdamW work better than Adam + L2 weight decay for transformers?