Deep Learning · section 18 of 19

Generative models

4 interview questions on generative models, each answered in full. Free to read, no account needed.

What is mode collapse in GANs and how do you mitigate it?

hard

$Generator maps most latents to a few output modes because those fool the discriminator well — trained distribution becomes much less diverse than the data.$
$Symptoms: repeated outputs, poor coverage of the true distribution.$
$Fixes: minibatch discrimination (D sees a batch and can penalize identical outputs), unrolled GAN (train D a few extra steps), WGAN-GP loss for smoother gradients, spectral norm on D, feature matching, or diffusion models (which don't suffer from mode collapse).$

hard

$Forward (q): gradually add Gaussian noise to the data over T steps until it becomes ~ N(0, I).$
$Fixed, no learning.$
$x_{t}\; = \;\mathrm{sqrt}(\alpha_{\mathrm{bar}}t) \cdot x_{0}\; + \;\mathrm{sqrt}(1\; - \;\alpha_{\mathrm{bar}}t) \cdot \epsilon$
$(p{\theta})$
$E_{t}$
$At sampling, start from Gaussian noise and denoise for T (or fewer, with DDIM) steps.$

#diffusion#generativePermalink & quiz →

hard

$DDIM (Song 2020) formulates a deterministic non-Markovian reverse process that shares the same training objective as DDPM but allows sampling with far fewer steps (e.g., 50 instead of 1000) with minimal quality loss.$
$Also enables deterministic sampling (identical noise \to identical output), which is essential for tasks like image-to-image editing, inversion, and reproducibility.$
$Foundation for modern samplers (Euler, Heun, DPM-Solver++).$

#diffusion#generativePermalink & quiz →

hard

$Train the model to accept both conditional (c) and unconditional (\emptyset, e.g., empty prompt) inputs, by dropping the condition 10-20% of the time.$
$\mathrm{eps}_{\mathrm{guided}}\; = \;\mathrm{eps}$
$Standard trick in Stable Diffusion / Imagen — biggest single quality lever after model size.$

#diffusion#generativePermalink & quiz →