EasyDeepLearn
Deep Learning · section 15 of 19

Losses for deep learning

11 interview questions on losses for deep learning, each answered in full. Free to read, no account needed.

Write the Huber loss and explain when to use it.

medium
  • L_δ(y,  )  =  0.5⋅(y−)2(y, \;)\; = \;0.5 \cdot (y - )^{2} if  ∣y\mathrm{if}\; \mid y-ŷ| ≤ δ, else δ⋅(∣y−∣  −  0.5⋅δ){\delta} \cdot ( \mid y - \mid \; - \;0.5 \cdot {\delta}).
  • Quadratic for small errors (smooth, MSE-like gradients), linear for large errors (robust to outliers). δ ~ 1-2*σ of the noise.
  • Common in bounding-box regression (Smooth L1 with δ=1 in R-CNN family) and robust regression.
#lossesPermalink & quiz →

What is log-cosh loss and its advantage over Huber?

medium
  • L(y, ŷ) = log(cosh(y - ŷ)) ≈ 0.5⋅(y−)20.5 \cdot (y - )^{2} for small errors and  ∣y\mathrm{and}\; \mid y-ŷ| - log⁡(2)\operatorname{log}(2) for large — same behavior as Huber but smooth everywhere (all derivatives exist).
  • No δ hyperparameter.
  • Slightly more expensive to compute.
  • Use it when you want Huber's outlier-robustness without tuning δ, and gradient smoothness matters (e.g., second-order optimizers).
#lossesPermalink & quiz →

Write binary cross-entropy for probability p̂ and label y ∈ {0, 1}.

easy
  • L(y, p̂) = -[y*log(p̂) + (1-y)*log(1-p̂)].
  • Comes from the negative log-likelihood of a Bernoulli.
  • Always non-negative, zero iff p̂=y ∈ {0,1}.
  • In practice, apply the numerically-stable BCEWithLogitsLoss (input is logits, sigmoid + BCE fused with log-sum-exp for stability) — never compute sigmoid then log separately in fp16.
#lossesPermalink & quiz →

Multi-class cross-entropy vs one-vs-rest — pros and cons?

medium
  • Cross-entropy on softmax: mutually exclusive classes, single loss, joint probability calibration → standard for classification.
  • One-vs-rest with K sigmoids + BCE: allows multi-label, imbalanced-per-label handling, per-class thresholding at inference.
  • Cross-entropy is more parameter-efficient for many classes; OVR is essential for multi-label but gives up joint calibration.
#lossesPermalink & quiz →

Write the basic contrastive loss and when it's used.

medium
  • L  =  y  ⋅  d2  +  (1−y)  ⋅  max⁡(margin  −  d,  0)2L\; = \;y\; \cdot \;d^{2}\; + \;(1 - y)\; \cdot \;\operatorname{max}(\mathrm{margin}\; - \;d, \;0)^{2}, where y=1 for a positive pair (same class) and y=0 for a negative pair, d is a distance in the embedding space.
  • Pulls positives together, pushes negatives apart at least by a margin.
  • Used in Siamese nets, verification (faces, signatures), and metric learning.
  • Modern SSL uses info-NCE (below) instead.
#losses#contrastive#self-supervisedPermalink & quiz →

What is triplet loss and its main challenge?

hard
  • L(a, p, n) = max(d(a, p) - d(a, n) + margin, 0), where a=anchor, p=positive, n=negative.
  • Wants the anchor closer to positives than to negatives by at least the margin.
  • Main challenge: 'triplet mining' — most random triplets are easy (already satisfied), so you must actively sample semi-hard / hard triplets to keep training progressing.
  • Standard in face recognition (FaceNet).
  • Superseded in most tasks by info-NCE / softmax-based losses.
#losses#contrastivePermalink & quiz →

Write info-NCE / NT-Xent and its role in SSL.

hard
  • L(x,  x+,  x−)L(x, \;x^{+}, \;x^{-}) = -log⁡[exp⁡(sim(x,  x+)  /  τ)  /  (exp⁡(sim(x,  x+)  /  τ)  +  Σ  exp⁡(sim(x,  x−)  /  τ))]\operatorname{log}[\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; / \;(\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; + \;{\Sigma}\;\operatorname{exp}(\mathrm{sim}(x, \;x^{-})\; / \;{\tau}))], sim = cosine similarity, τ = temperature.
  • Treats contrastive learning as a classification problem: 'which of these candidates is the positive?'.
  • Foundation of SimCLR, MoCo, CLIP, GPT-style retrieval.
  • Larger negative batches → richer learning signal.
#losses#contrastive#self-supervisedPermalink & quiz →

What is the standard reconstruction loss for autoencoders?

easy
  • MSE (or L1) between input and reconstruction for continuous data.
  • Binary cross-entropy per pixel for images normalized to [0, 1] (treats each pixel as a Bernoulli).
  • VAEs add a KL divergence between the posterior over latents and a prior — the ELBO objective.
  • Modern generative models use perceptual losses (feature-space MSE) or adversarial losses for sharper reconstructions.
#losses#generativePermalink & quiz →

Write the vanilla GAN's minimax objective.

hard
  • min⁡G\operatorname{min}_{G} max⁡D\operatorname{max}_{D} Ex[log⁡  D(x)]  +  Ez[log⁡(1  −  D(G(z)))]E_{x}[\operatorname{log}\;D(x)]\; + \;E_{z}[\operatorname{log}(1\; - \;D(G(z)))].
  • D tries to output 1 for real, 0 for fake; G tries to make D output 1 on its fakes.
  • In practice, G is trained with the 'non-saturating' loss  −Ez[log⁡  D(G(z))]\mathrm{loss}\; - E_{z}[\operatorname{log}\;D(G(z))] (stronger gradients when D is confident).
  • WGAN replaces this with a Wasserstein-distance-based loss for better stability.
#losses#gan#generativePermalink & quiz →

What does WGAN + gradient penalty change vs vanilla GAN?

hard
  • WGAN uses the Wasserstein distance approximation: max⁡D\operatorname{max}_{D} E[D(x)] - E[D(G(z))] where D (the 'critic') must be 1-Lipschitz.
  • Original WGAN clipped weights (crude).
  • WGAN-GP (Gulrajani 2017) enforces Lipschitz softly with a gradient penalty λ  ⋅  (∣∣∇x  D(x)∣∣2  −  1)2{\lambda}\; \cdot \;( \mid \mid \nabla x\;D(x) \mid \mid _{2}\; - \;1)^{2} on interpolated points x̂.
  • Result: stable training, meaningful loss curve (correlated with sample quality), fewer mode-collapse issues.
#gan#generative#lossesPermalink & quiz →

Write the VAE ELBO and explain each term.

hard
  • ELBO(x)  =  Eq(z∣x)\mathrm{ELBO}(x)\; = \;E_{q}(z \mid x)[log⁡  p(x∣z)]  −  KL⁡(q(z∣x)  ∣∣  p(z))[\operatorname{log}\;p(x \mid z)]\; - \;\operatorname{KL}(q(z \mid x)\; \mid \mid \;p(z)).
  • Reconstruction term: how well the decoder reproduces x from a latent sampled by the encoder.
  • KL term: pushes the encoder's posterior q(z∣x)q(z \mid x) toward the prior p(z) (usually N(0, I)).
  • Maximizing ELBO ≤ log p(x).
  • The KL acts as regularizer — without it, VAE collapses to an autoencoder. β-VAE weights the KL term for disentanglement.
#vae#generative#lossesPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz