EasyDeepLearn
Unsupervised Learning · section 7 of 9

Autoencoders & VAEs

34 interview questions on autoencoders & vaes, each answered in full. Free to read, no account needed.

What is self-supervised learning and why does it matter?

medium
  • Self-supervised learning creates labels from the data itself via pretext tasks (masked language modeling, next-token prediction, contrastive views of images) — no manual annotation.
  • It matters because it lets us pretrain huge models on unlabeled data and then transfer to downstream tasks with little labeled data.
  • It's the engine behind BERT, GPT, SimCLR, CLIP, MAE, DINO.
#representation-learningPermalink & quiz →

How does contrastive learning work?

hard
  • Create two augmented views of the same example (positive pair) and treat other examples as negatives.
  • Train an encoder so positives are close in embedding space and negatives are far, using a loss like InfoNCE.
  • Examples: SimCLR, MoCo, CLIP (contrasts image with text).
  • Produces general-purpose embeddings useful for many downstream tasks.
#representation-learningPermalink & quiz →

What is an autoencoder and what are the common variants?

medium
  • An autoencoder learns to compress input x through a bottleneck and reconstruct it.
  • Variants: denoising AE (train to reconstruct clean from noisy input), sparse AE (bottleneck via sparsity penalty), variational AE (probabilistic latent, generative), masked AE (mask patches, reconstruct — used in vision pretraining).
  • Uses: dimensionality reduction, denoising, anomaly detection, pretraining.
#representation-learning#deep-learningPermalink & quiz →

Autoencoder variants — quick tour.

medium
  • (1) Undercomplete AE: bottleneck < input dim → compression.
  • (2) Denoising AE: reconstruct clean from noised input → robustness / representation.
  • (3) Sparse AE: L1 or KL penalty on activations → sparse code.
  • (4) Contractive AE: penalize ||∂h/∂x||_F → local invariance.
  • (5) VAE: probabilistic latent, generative.
  • (6) Masked AE (MAE, He et al. 2022): mask 75% of patches, reconstruct — SOTA vision SSL.
#representation-learning#deep-learningPermalink & quiz →

Derive the VAE ELBO.

hard
  • log p(x) = log ∫ p(x∣z)p(x \mid z) p(z) dz ≥ Eq(z∣x)[log⁡  p(x∣z)]  −  KL⁡(q(z∣x)  ∣∣  p(z))E_{q(z \mid x)}[\operatorname{log}\;p(x \mid z)]\; - \;\operatorname{KL}(q(z \mid x)\; \mid \mid \;p(z)) (Jensen).
  • ELBO decomposes into reconstruction (log-likelihood of x given z) minus KL to prior.
  • Encoder q_φ(z∣x)(z \mid x) parameterized as Gaussian (μ,  σ2)({\mu}, \;{\sigma}^{2}); decoder p_θ(x∣z)(x \mid z).
  • Reparameterization trick: z = μ + σ ⊙ ε with ε ~ N(0, I) → backprop through sample.
  • Optimize both φ, θ jointly.
#representation-learning#deep-learningPermalink & quiz →

VAE vs plain autoencoder — the key advantages.

medium
  • (1) Probabilistic latent → generative (sample z ~ p(z), decode → new x).
  • (2) Regularized latent space by KL to prior → smooth interpolation, meaningful arithmetic.
  • (3) Calibrated uncertainty.
  • Weaknesses: VAE samples are typically blurrier than GAN samples (mean-of-modes issue), latent may collapse (posterior collapse), harder to train.
  • Modern hybrid: VQ-VAE, β-VAE, hierarchical VAE (NVAE) address these.
#representation-learning#deep-learningPermalink & quiz →

β-VAE — what does the β hyperparameter do?

hard
  • Modifies ELBO to Eq[log⁡  p(x∣z)]  −  β  ⋅  KL⁡(q  ∣∣  p)E_{q}[\operatorname{log}\;p(x \mid z)]\; - \;{\beta}\; \cdot \;\operatorname{KL}(q\; \mid \mid \;p). β > 1 pushes KL harder → more independent latent factors → disentangled representations (each latent dim captures one factor: pose, color, size).
  • Cost: worse reconstruction.
  • Trade-off between disentanglement (large β) and fidelity (small β).
  • Foundation of disentangled representation learning; extended by FactorVAE, β-TCVAE.
#representation-learning#deep-learningPermalink & quiz →

VQ-VAE — the core idea.

hard
  • Discrete latent codes: encoder output quantized to nearest code in a learned codebook (like k-means).
  • Straight-through gradient estimator for backprop.
  • Uses: (1) discrete-latent generative modeling (audio in WaveNet-style, video in Sora / VideoPoet), (2) tokenizer for text-image (DALL-E), (3) speech (Wav2Vec 2).
  • Enables autoregressive priors on discrete latents.
  • Foundation of many modern multi-modal generative models.
#representation-learning#deep-learningPermalink & quiz →

SimCLR — recipe.

medium
  • Contrastive framework for visual SSL: (1) two random augmentations of each image → positive pair, (2) encoder + projection head, (3) NT-Xent loss (InfoNCE variant): pull positives together, push all other batch items apart.
  • Needs LARGE batch size (~4096) for enough negatives.
  • Removes projection head at downstream time.
  • Started the modern SSL for vision revolution; superseded by SimSiam / DINO / MAE.
#representation-learningPermalink & quiz →

MoCo (Momentum Contrast) — how does it enable smaller batches?

hard
  • Maintain a queue of negatives from previous batches instead of relying on current batch only.
  • Momentum encoder: EMA of the main encoder → consistent keys in queue as encoder updates slowly.
  • Enables SimCLR-level performance with batches of 256 instead of 4096.
  • MoCo v2 / v3 improved augmentations, projection head, ViT backbone.
  • Foundation of large-scale efficient SSL.
#representation-learningPermalink & quiz →

BYOL — how does it avoid the need for negatives?

hard
  • Two networks: online (learnable) and target (EMA of online).
  • Feed different augmentations to each; online must predict target's representation via a predictor head.
  • No negatives, no contrastive loss — momentum + asymmetric predictor prevents collapse.
  • Surprising and empirical: BYOL matches SimCLR without negatives.
  • Successor SimSiam removes even the momentum target (relies only on stop-gradient).
#representation-learningPermalink & quiz →

SimSiam — what makes it minimal?

hard
  • Removes EMA target: both branches share weights.
  • Uses only stop-gradient on one branch and an asymmetric predictor head.
  • Still matches BYOL / SimCLR.
  • Chen & He 2021 showed non-collapse is achieved purely by stop-gradient asymmetry — huge simplification.
  • Foundation of understanding why SSL works: asymmetric optimization dynamics, not contrast, is the key.
#representation-learningPermalink & quiz →

DINO — self-distillation with no labels.

hard
  • Student network learns to match teacher (EMA of student) on different augmentations of same image.
  • Uses ViT backbone → produces amazing self-attention maps (unsupervised object segmentation emerges!).
  • Centering + sharpening on teacher output prevents mode collapse.
  • DINOv2 (Meta 2023) scaled it to 1B parameters + 142M images → foundation vision encoder rivaling CLIP.
#representation-learningPermalink & quiz →

MAE (Masked Autoencoder) — He et al. 2022.

medium
  • Mask 75% of image patches, encode only visible patches, decode all patches (mask tokens).
  • Loss: MSE on masked pixel patches.
  • Encoder never sees mask tokens → fast pretraining.
  • Simple, scalable, competitive with contrastive methods.
  • Same idea as BERT for text.
  • Vision SOTA circa 2022-2023 alongside DINO.
  • Foundation of many modern vision transformers (SAM's encoder uses MAE pretraining).
#representation-learning#deep-learningPermalink & quiz →

CLIP — how does contrastive image-text training work?

medium
  • Train two encoders (image ViT + text Transformer) so that matching (image, caption) pairs are close in shared embedding space, mismatched pairs far.
  • InfoNCE loss over batch.
  • Trained on 400M image-caption pairs from web.
  • Result: zero-shot image classification via 'a photo of a {class}' prompt, cross-modal retrieval, foundation for DALL-E / Stable Diffusion.
  • Rewrote vision-language modeling; superseded by SigLIP, EVA-CLIP, SAM 2.
#representation-learningPermalink & quiz →

JEPA (I-JEPA, V-JEPA) — LeCun's alternative to generative SSL.

hard
  • Predict abstract features (not pixels) of masked regions in embedding space.
  • Sidesteps pixel-level detail (irrelevant + wasteful) and models context predictively.
  • I-JEPA (images 2023): predicts feature representation of masked target block from context.
  • V-JEPA (video 2024): same for video.
  • Faster + more semantic representations than MAE.
  • LeCun's flagship 'world model' architecture.
#representation-learning#deep-learningPermalink & quiz →

InfoNCE loss — formula and intuition.

hard
  • L = -log [exp⁡(s(x,  x+)/τ)  /  Σj  exp⁡(s(x,  xj)/τ)][\operatorname{exp}(s(x, \;x^{+}) / {\tau})\; / \;{\Sigma}_{j}\;\operatorname{exp}(s(x, \;x_{j}) / {\tau})] — softmax over one positive and many negatives, temperature τ.
  • Cross-entropy that treats classification 'which of N is the positive?'.
  • Lower bound on mutual information between x and x+x^{+}.
  • Standard loss for contrastive SSL (SimCLR, CLIP, MoCo, etc.).
  • Temperature τ controls concentration: low τ → sharp, hard-negative-focused.
#representation-learning#theoryPermalink & quiz →

Why are augmentations so important in contrastive SSL?

medium
  • Augmentations define the invariances the encoder learns — 'same-object under transformation' becomes the training signal.
  • Strong compositions (crop + color jitter + Gaussian blur) work best (Chen et al. 2020).
  • Too weak → trivial invariances; too strong → destroys semantic content.
  • Rule: task-relevant invariances.
  • Cross-modal alignment (CLIP) uses text as 'super-augmentation'.
#representation-learningPermalink & quiz →

Word2Vec skip-gram — objective and training.

medium
  • For each target word, predict surrounding context words (window size ~5).
  • Softmax over full vocabulary too expensive → negative sampling: for each positive (target, context) pair, sample k random 'negative' words and train binary logistic.
  • Learns dense word vectors capturing distributional semantics.
  • Levy & Goldberg 2014: mathematically equivalent to implicit factorization of shifted PMI matrix.
#representation-learning#nlpPermalink & quiz →

GloVe vs Word2Vec.

medium
  • GloVe (Pennington 2014): explicit weighted matrix factorization of log co-occurrence counts.
  • Objective: log⁡(Xij)\operatorname{log}(X_{\mathrm{ij}}) ≈ wiw_{i}' wj  +  bi  +  bjw_{j}\; + \;b_{i}\; + \;b_{j}.
  • Batch training on global statistics (vs Word2Vec's per-window streaming).
  • Similar quality to Word2Vec; GloVe often better on analogy tasks, Word2Vec on similarity.
  • Modern: both superseded by contextual embeddings (BERT / sentence-transformers).
  • Still cited in interview questions.
#representation-learning#nlpPermalink & quiz →

Word embedding analogies — why do they work?

hard
  • king - man + woman ≈ queen.
  • Emerges because embeddings capture distributional differences.
  • Actually more fragile than the meme suggests: only works for common frequent analogies, and typical evaluations use cosine similarity + exclusion of query words.
  • Modern contextual embeddings capture analogy implicitly through generation.
  • Interview trap: don't overstate this as evidence of 'reasoning'.
#representation-learning#nlpPermalink & quiz →

Sentence-BERT — how does it improve on BERT for retrieval?

medium
  • BERT's [CLS] token is not a great semantic sentence vector out of the box.
  • SBERT fine-tunes BERT with a siamese architecture on NLI + STS: same encoder on two sentences, minimize distance for paraphrases, maximize for contradictions.
  • Cosine similarity ≈ semantic similarity.
  • Enables efficient sentence retrieval (encode once, cosine at query time), replaced by newer E5 / BGE / OpenAI text-embedding-3 for scale, but SBERT is still baseline.
#representation-learning#nlpPermalink & quiz →

Node2Vec — how does it learn graph node embeddings?

medium
  • Random walks starting from each node → treat walks as 'sentences', apply Word2Vec skip-gram → embed nodes.
  • Biased walks (p, q) control breadth (BFS-like) vs depth (DFS-like): controls whether embeddings capture structural equivalence or community proximity.
  • Uses: link prediction, node classification, community detection preprocessing.
  • Foundational shallow graph embedding; modern replacement: GNNs (GraphSAGE, GAT).
#representation-learning#applicationsPermalink & quiz →

Modern graph representation learning — GNNs.

hard
  • Graph Neural Networks aggregate neighbor features iteratively: hvl+1  =  UPDATEh_{v}^{l + 1}\; = \;\mathrm{UPDATE}(hvlh_{v}^{l}, AGG(hul    u  ∈  N(v))\mathrm{AGG}(h_{u}^{l}\;\;u\; \in \;N(v))).
  • Variants: GraphSAGE (sampled aggregation), GAT (attention weights), GCN (spectral), MPNN (message passing).
  • Trained supervised, semi-sup, or self-supervised (GraphCL, BGRL).
  • Uses: recommendation, drug discovery, molecule property prediction, fraud rings.
#representation-learning#deep-learningPermalink & quiz →

Self-supervised graph learning — approaches.

hard
  • (1) Node-level: contrastive on augmented views (drop edges/nodes/features), GraphCL.
  • (2) Predict masked node attributes (MaskGAE).
  • (3) Motif / structure prediction.
  • (4) Bootstrap approaches (BGRL, analog of BYOL for graphs).
  • Used to pretrain GNNs on unlabeled graphs before fine-tuning on labels.
  • Standard in drug discovery / molecular property pipelines (huge unlabeled compound libraries).
#representation-learning#applicationsPermalink & quiz →

Self-supervised audio — wav2vec 2 and HuBERT.

hard
  • Wav2Vec 2 (Facebook 2020): quantize audio features → mask spans → predict quantized targets via contrastive loss.
  • HuBERT: similar but uses clustered representations as pseudo-labels (BERT-style masked prediction).
  • Both dominated pretraining for ASR — supervised fine-tuning with ~10 hours of labeled speech reaches WER competitive with 1000+ hours of pure supervised training.
  • Foundation of Whisper's competitors and many production speech systems.
#representation-learning#applicationsPermalink & quiz →

How are multimodal embeddings unified across text / image / audio?

hard
  • Approaches: (1) CLIP-style paired training (image ↔ text).
  • (2) ImageBind (Meta 2023): 6 modalities aligned through image as the 'bridge' — no need for all-pair data.
  • (3) Whisper + CLIP for text-audio alignment.
  • (4) LLaVA-style: project image encoder into LLM embedding space.
  • Common thread: contrastive alignment + a bridge modality.
  • Foundation of multimodal LLMs.
#representation-learning#applicationsPermalink & quiz →

Scaling laws for self-supervised pretraining.

hard
  • Downstream performance improves as a power law in (data, params, compute).
  • MAE / CLIP / DINOv2 / vision transformers all show emergent capabilities at scale (rare classes, zero-shot).
  • Diminishing returns: 10× more compute typically gives ~linear gain in loss.
  • Data quality matters more than quantity at very large scale (DINOv2 uses curated 142M images vs LAION's 5B).
  • Modern trend: careful data selection + scale.
#representation-learning#theoryPermalink & quiz →

Mode / representation collapse in SSL — what and why?

hard
  • Encoder outputs the same (or trivially degenerate) representation for all inputs → useless embeddings.
  • Root cause: shortcut solutions minimizing loss without capturing content.
  • Prevention: (1) contrastive negatives, (2) asymmetry (stop-gradient + predictor, BYOL / SimSiam), (3) centering + sharpening (DINO), (4) variance-covariance regularization (VICReg, Barlow Twins), (5) predictor bottleneck.
  • Understanding when/why collapse is avoided is an open research direction.
#representation-learning#theoryPermalink & quiz →

VICReg / Barlow Twins — non-contrastive SSL via covariance regularization.

hard
  • Barlow Twins: cross-correlation of two augmented view embeddings should be identity (diagonal = 1, off-diag = 0 → invariance + de-correlation).
  • VICReg extends with three terms: invariance (MSE between views), variance (each dim has SD > threshold), covariance (off-diag = 0).
  • No negatives, no momentum encoder — just clean regularization.
  • Elegant, competitive with contrastive methods.
#representation-learningPermalink & quiz →

Fine-tuning vs linear probe vs prompt-tuning — when do you use each?

medium
  • (1) Linear probe: small labeled data + strong SSL encoder → freeze encoder, add linear head.
  • Fast, tiny compute.
  • (2) Fine-tuning: enough labeled data + task shift → unfreeze last k layers or full network.
  • (3) LoRA / adapters: parameter-efficient, when full fine-tune too expensive.
  • (4) Prompt-tuning: LLM-specific — learn soft prompts instead of weights.
  • Rule: start with linear probe as baseline; escalate to fine-tune only if needed.
#representation-learning#applicationsPermalink & quiz →

What is a 'foundation model' in the unsupervised sense?

medium
  • Large model pretrained on massive unlabeled data via SSL → generalizes to many downstream tasks with little or no fine-tuning.
  • Examples: GPT / Llama (text), CLIP / DINOv2 (vision), SAM (segmentation), Whisper (speech), ImageBind (multimodal).
  • Bommasani et al. 2021 coined the term.
  • Enables the modern 'pretrain once, adapt many' paradigm — economically transformative in industry.
#representation-learning#applicationsPermalink & quiz →

Interview: 'you have unlabeled images — which SSL method should you use?'

hard
  • Ask: (1) Compute budget?
  • MAE is fastest to train.
  • Contrastive (SimCLR) needs huge batches / MoCo needs momentum.
  • (2) Downstream task type?
  • Classification → most methods work; segmentation → DINO or MAE better (spatial features); retrieval → contrastive (SimCLR, CLIP if paired text).
  • (3) Data volume?
  • Small → use pretrained + fine-tune, don't retrain SSL.
  • Large → DINOv2 or MAE.
  • (4) Domain?
  • Medical / satellite / etc — start from ImageNet-pretrained + continue SSL.
#interview#representation-learningPermalink & quiz →

Interview: 'when should you use a VAE vs GAN vs diffusion for generation?'

hard
  • (1) VAE: fast, latent space useful for interpolation + representation, but blurry samples.
  • Best when interpretable latent matters (drug design, molecule generation).
  • (2) GAN: sharp samples, fast inference, but training unstable + mode collapse.
  • Best for face generation, image editing (StyleGAN).
  • (3) Diffusion: SOTA quality, controllable, but slow sampling (50-1000 steps → mitigated by DDIM, distillation, Flow Matching).
  • Best default for images / video / audio generation in 2024+.
#interview#representation-learning#deep-learningPermalink & quiz →

Practise Unsupervised Learning

All 9 sectionsFlashcardsQuiz