MAE (Masked Autoencoder) — He et al. 2022.
mediumAnswer
- Mask 75% of image patches, encode only visible patches, decode all patches (mask tokens).
- Loss: MSE on masked pixel patches.
- Encoder never sees mask tokens → fast pretraining.
- Simple, scalable, competitive with contrastive methods.
- Same idea as BERT for text.
- Vision SOTA circa 2022-2023 alongside DINO.
- Foundation of many modern vision transformers (SAM's encoder uses MAE pretraining).
Check yourself — multiple choice
- Random
- Mask 75% patches, encode visible only, decode all; MSE on masked patches; simple, scalable; BERT-for-vision; SAM uses MAE pretraining
- Same as SimCLR
- Not real
MAE: mask 75% patches + reconstruct; BERT-for-vision; simple scalable SSL.
#representation-learning#deep-learning
Practise Unsupervised Learning
214 interview questions in this topic.