EasyDeepLearn
Deep Learning · section 8 of 19

Regularization

9 interview questions on regularization, each answered in full. Free to read, no account needed.

What is DropConnect and how is it different from Dropout?

hard
  • Dropout zeros activations; DropConnect zeros weights (elements of the weight matrix) — i.e., randomly severs individual connections.
  • More granular regularization but harder to implement efficiently on GPUs.
  • Occasionally used in specialized architectures but Dropout dominates in practice due to speed.
#regularizationPermalink & quiz →

Where is dropout typically inserted in a transformer block?

medium
  • (1) On attention weights (after softmax), (2) on attention output projections, (3) on MLP hidden and output activations, (4) on residual sums (dropout after each residual add — 'residual dropout').
  • Rates: ~0.1 for pretraining (BERT), sometimes 0.0 for LLM pretraining (large models overfit less).
  • Fine-tuning usually adds dropout on the classifier head.
#regularization#transformersPermalink & quiz →

What is label smoothing and why does it help?

easy
  • Replace one-hot target [0, ..., 1, ..., 0] with (1-α)*one-hot + α/K (uniform), typically α=0.1.
  • Prevents the network from becoming over-confident: it can't drive the softmax logit for the true class to infinity because the target isn't exactly 1.
  • Improves calibration, reduces overconfidence, and often improves accuracy 0.5-1% on classification benchmarks.
  • Standard in ImageNet training and NMT.
#regularization#lossesPermalink & quiz →

How does Mixup work?

medium
  • Sample two examples (xi,  yi)(x_{i}, \;y_{i}), (xj,  yj)(x_{j}, \;y_{j}) and a mixing coefficient λ ~ Beta(α, α) with α ~ 0.2.
  • Train on the interpolation: xnew  =  λ⋅xi  +  (1−λ)⋅xjx_{\mathrm{new}}\; = \;{\lambda} \cdot x_{i}\; + \;(1 - {\lambda}) \cdot x_{j}, ynew  =  λ⋅yi  +  (1−λ)⋅yjy_{\mathrm{new}}\; = \;{\lambda} \cdot y_{i}\; + \;(1 - {\lambda}) \cdot y_{j}.
  • Forces linearity between examples — smoother decision boundaries, better calibration, robustness to adversarial noise.
  • Widely used in vision.
  • Doesn't work as well on text without variants.
#augmentation#mixup#regularizationPermalink & quiz →

What is Cutout / Random Erasing?

easy
  • During training, mask a random rectangular region of the input image with zeros (Cutout) or random noise (Random Erasing).
  • Forces the network to look at the whole image rather than one salient region — improves robustness to occlusion.
  • Simple, cheap, effective.
  • Standard in vision recipes since 2017; combines well with Mixup / CutMix.
#augmentation#regularizationPermalink & quiz →

Why is data augmentation effectively free regularization?

easy
  • Every augmentation is a prior about invariances the true function satisfies (translations of a cat are still a cat).
  • Applying augmentations enlarges the effective training distribution without labeling cost.
  • Forces the model to be invariant to those transformations, which reduces overfitting to specific instances.
  • Free lunch in vision; needs task-specific care in text and tabular.
#augmentation#regularizationPermalink & quiz →

What is consistency regularization?

medium
  • Enforce that two augmented views of the same unlabeled input produce similar predictions: Lcons  =  DL_{\mathrm{cons}}\; = \;D(f(aug1(x)), f(aug2(x))) with D = KL or MSE.
  • Powers modern semi-supervised methods (FixMatch, Mean Teacher, MixMatch) and self-supervised learning (SimCLR, MoCo, BYOL).
  • Cornerstone technique when labeled data is scarce.
#regularization#self-supervised#augmentationPermalink & quiz →

What is Manifold Mixup?

hard
  • Variant of Mixup (Verma et al., 2019): at each step, pick a random hidden layer k and linearly interpolate the hidden representations of two examples at that layer (rather than at the input).
  • Encourages smoother hidden representations, further improves calibration and robustness.
  • Slightly more expensive per step, marginal gains over vanilla Mixup on top-tier vision benchmarks.
#mixup#regularizationPermalink & quiz →

Why does dropout play a smaller role in large language model training than in older vision models?

hard
  • Because the regularization pressure comes from the data instead.
  • When a model sees each token roughly once, on a corpus far larger than its parameter count, memorization is not the binding constraint, so the variance reduction dropout provides is not needed and its added gradient noise mostly slows convergence.
  • Older vision models made many passes over a small labelled set, where overfitting was the dominant failure and dropout genuinely helped.
  • Modern large-scale pretraining therefore uses little or no dropout, relying on data volume, weight decay and early stopping in the form of a single epoch.
  • Dropout returns during fine-tuning on small task datasets, where the classic overfitting regime is back.
#regularization#transformers#fine-tuningPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz