EasyDeepLearn
All categoriesStudy
Lesson

Deep Learning

214 concepts~321 min read19 sections

Neural networks, training tricks, CNNs, RNNs, transformers and everything in between.

Interview flashcardsTake the quiz
Filter by difficulty
Filter by concept tag

Introduction

Deep learning is what happens when you stack differentiable layers deep enough — and add enough data, compute, and clever tricks — for gradient descent to discover useful representations by itself.

Interview questions here fall into three buckets: (1) the numerical and optimization tricks that make training work (initialization, normalization, activations, learning-rate schedules); (2) architectures with useful inductive biases (CNNs for images, transformers for sequences); (3) understanding *why* certain choices dominate the modern stack. This chapter walks through the essentials without hand-waving.

01

Activations & fundamentals

25 concepts

ReLU vs sigmoid vs GELU — when do you use each?

easy
  • Sigmoid/tanh saturate and cause vanishing gradients — avoid in hidden layers of deep nets, keep sigmoid for a binary output.
  • ReLU is the default hidden activation: fast, sparse, but suffers from 'dying ReLU' (permanently zero neurons).
  • Leaky ReLU / ELU fix that.
  • GELU is a smooth approximation of ReLU and is the standard in transformers (BERT, GPT).
  • SiLU/Swish is also popular in modern vision and LLMs.
Permalink

In one sentence, what is backpropagation?

easy
  • Backpropagation is the reverse-mode automatic differentiation algorithm that efficiently computes gradients of a scalar loss with respect to every parameter of a neural network by traversing the computation graph backwards, applying the chain rule at each node.
Permalink

State the universal approximation theorem in one sentence and its practical caveat.

medium
  • A feedforward network with a single hidden layer of finite width can approximate any continuous function on a compact domain to arbitrary accuracy — but the theorem says nothing about width required, training feasibility, or generalization.
  • In practice depth is much more parameter-efficient than width, which is why deep networks dominate.
Permalink

Why does stacking linear layers without nonlinearity collapse to a single linear layer?

easy
  • The composition of linear maps is a linear map: W2W_{2} (W1  x  +  b1)  +  b2  =  (W2  W1)(W_{1}\;x\; + \;b_{1})\; + \;b_{2}\; = \;(W_{2}\;W_{1}) x  +  (W2  b1  +  b2)x\; + \;(W_{2}\;b_{1}\; + \;b_{2}).
  • No matter how many layers you stack, the effective function is W x + b.
  • Non-linear activations (ReLU, GELU, tanh) break this and unlock representational power that grows with depth.
Permalink

Depth vs width — which do you scale first?

medium
  • Depth compounds representational capacity exponentially in the number of piecewise-linear regions for ReLU networks, while width only grows it polynomially.
  • So depth is usually the better initial investment for representation.
  • However, very deep networks are harder to optimize — need normalization, residual connections, and careful init.
  • Modern architectures (transformers, ConvNeXt) balance both via 'compound scaling'.
Permalink

Why can't you initialize a neural net with all zeros?

easy
  • Symmetric weights mean every neuron in a layer computes the same output and receives the same gradient — the layer effectively has one neuron.
  • Training never breaks the symmetry.
  • Break it with any random init (Xavier / He / uniform).
  • Biases can be initialized to zero because they aren't multiplied together; only the weight matrices need randomness.
Permalink

What is a computation graph and why does autograd care?

easy
  • A DAG whose nodes are operations and edges track data + gradient dependencies.
  • During the forward pass, each op records its inputs so its backward function can be called.
  • Reverse-mode autodiff (backprop) traverses the graph from the loss backwards, applying the chain rule at each node.
  • PyTorch builds it dynamically per iteration; TensorFlow (TF1) and JAX build it statically.
Permalink

What is 'dying ReLU' and how do you fix it?

medium
  • A ReLU neuron that always outputs zero has a zero gradient everywhere and stops learning permanently.
  • Causes: large negative bias, extreme learning rate, poor init.
  • Fixes: Leaky ReLU or ELU (nonzero slope on the negative side), He initialization, lower learning rate, better data normalization, or use GELU / SiLU which are smooth and don't fully saturate to zero.
Permalink

What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?

medium
  • SiLU⁡(x)  =  x  ⋅  sigmoid⁡(x)\operatorname{SiLU}(x)\; = \;x\; \cdot \;\operatorname{sigmoid}(x).
  • GELU⁡(x)\operatorname{GELU}(x) ≈ x * Phi(x) — smooth version of ReLU using a Gaussian CDF.
  • Both are smooth and non-monotonic near zero, giving richer gradients than ReLU's hard elbow.
  • Empirically better on transformers and modern vision (ConvNeXt, EfficientNet).
  • GELU is standard in BERT / GPT / most LLMs; SiLU in vision (EfficientNet) and some LLMs (Llama, Mistral).
Permalink

What does temperature do in a softmax?

easy
  • softmax⁡(z  /  T)\operatorname{softmax}(z\; / \;T): T > 1 flattens the distribution (softer, higher entropy — used in distillation for teacher outputs), T < 1 sharpens toward one-hot (used in decoding to make choices more deterministic), T = 0 is argmax.
  • In LLM decoding, temperature is a knob for creativity vs determinism.
  • In distillation, teacher and student use matching high T so students learn full distribution info.
Permalink

Why compute softmax + cross-entropy jointly via log-sum-exp?

medium
  • Naive softmax⁡  =  exp⁡(zi)  /  sum(exp⁡(zj))\operatorname{softmax}\; = \;\operatorname{exp}(z_{i})\; / \;\mathrm{sum}(\operatorname{exp}(z_{j})) overflows for large logits.
  • Compute log⁡−softmax⁡(zi)  =  zi  −  log⁡sumexp⁡\operatorname{log} - \operatorname{softmax}(z_{i})\; = \;z_{i}\; - \;\operatorname{log}_{\mathrm{sum}}\operatorname{exp}(z) where log⁡sumexp⁡\operatorname{log}_{\mathrm{sum}}\operatorname{exp}(z) = max⁡(z)  +  log⁡\operatorname{max}(z)\; + \;\operatorname{log}(sum(exp⁡(z  −  max⁡(z))\operatorname{exp}(z\; - \;\operatorname{max}(z)))).
  • Numerically stable.
  • Cross-entropy then becomes -sum(target * log-softmax) — computed in one op (nn.CrossEntropyLoss in PyTorch), avoiding a separate softmax that could overflow before the log.
Permalink

How should the output head be designed for a regression task with a strictly positive target?

medium
  • Options: (1) predict log⁡(y)\operatorname{log}(y) and exponentiate — implicit positivity, natural for right-skewed targets; (2) apply exp⁡()\operatorname{exp}() or softplus at the output to guarantee positivity; (3) predict a mean of a Gamma/log-normal distribution via a GLM-style head.
  • Never use identity output with MSE if y can only be positive — the model will predict negatives during training which then contribute noise.
Permalink

How is the output head different for multi-label vs multi-class classification?

easy
  • Multi-class: one softmax head, one loss (categorical cross-entropy) — labels are mutually exclusive.
  • Multi-label: K independent sigmoid heads, one binary cross-entropy loss per label, summed — a document can have multiple tags simultaneously.
  • Do not use softmax for multi-label: it forces the K probabilities to sum to 1, which contradicts independence.
Permalink

Do you need bias terms in every layer of a modern neural network?

medium
  • Often no.
  • When a layer is followed by BatchNorm or LayerNorm with learned shift (beta), the bias is redundant — the norm's beta term absorbs it.
  • Standard practice: linear + norm → set bias=False on the linear.
  • Modern transformers frequently omit biases from Q/K/V projections and MLP layers when they use RMSNorm or LayerNorm — small memory / compute savings, no accuracy hit.
Permalink

How do you compute the parameter count of a linear layer and a convolution?

medium
  • Linear(in, out): in * out + out (weights + bias).
  • Conv2d(Cin,  Cout,  kernel=k,  stride=s,  pad=p)\mathrm{Conv2d}(C_{\mathrm{in}}, \;C_{\mathrm{out}}, \;\mathrm{kernel} = k, \;\mathrm{stride} = s, \;\mathrm{pad} = p): Cin  ⋅  Cout  ⋅  k  ⋅  k  +  CoutC_{\mathrm{in}}\; \cdot \;C_{\mathrm{out}}\; \cdot \;k\; \cdot \;k\; + \;C_{\mathrm{out}}.
  • Multi-head attention with hidden d and n heads: 4  ⋅  d24\; \cdot \;d^{2} (Q, K, V, out projections) + biases.
  • MLP block: 2  ⋅  d  ⋅  dffn2\; \cdot \;d\; \cdot \;d_{\mathrm{ffn}} (up + down projection).
  • Practical parameter count for a transformer layer ≈ 12  ⋅  d212\; \cdot \;d^{2} (attn  +  MLP  with  dffn  =  4d)(\mathrm{attn}\; + \;\mathrm{MLP}\;\mathrm{with}\;d_{\mathrm{ffn}}\; = \;4d).
Permalink

How do FLOPs scale for a forward pass through an MLP and a transformer?

hard
  • MLP layer(in, out) on batch B: 2 * B * in * out FLOPs.
  • Transformer forward (per token, per layer) ≈ 2  ⋅  nlayers  ⋅  d  ⋅  dffn2\; \cdot \;n_{\mathrm{layers}}\; \cdot \;d\; \cdot \;d_{\mathrm{ffn}} (MLP)  +  2  ⋅  nlayers  ⋅  d  ⋅  (d  +  dhead)(\mathrm{MLP})\; + \;2\; \cdot \;n_{\mathrm{layers}}\; \cdot \;d\; \cdot \;(d\; + \;d_{\mathrm{head}}) (attention  linear  projections)  +  2  ⋅  nlayers  ⋅  seq  ⋅  d(\mathrm{attention}\;\mathrm{linear}\;\mathrm{projections})\; + \;2\; \cdot \;n_{\mathrm{layers}}\; \cdot \;\mathrm{seq}\; \cdot \;d (attention softmax + matmuls).
  • For long context, attention dominates because it's O(seq2  ⋅  d)O(\mathrm{seq}^{2}\; \cdot \;d).
  • Approximation: forward FLOPs ≈ 2 * N (params) per token — thus training a model with N params on T tokens takes ~6 * N * T FLOPs (forward + backward + optimizer).
Permalink

Why does tanh cause vanishing gradients in deep nets?

medium
  • tanh'(x)  =  1  −  tanh2(x)(x)\; = \;1\; - \;\mathrm{tanh}^{2}(x) ≤ 1, and is close to 1 only near x = 0.
  • Deep nets keep multiplying such derivatives during backprop — the product shrinks toward 0.
  • With He/Xavier init, activations at each layer land in the saturating region (∣x∣  >  2)( \mid x \mid \; > \;2), where the derivative is near zero.
  • Fix: use ReLU/GELU which don't saturate on the positive side, plus normalization to keep activations in the useful range.
Permalink

What is a forward hook in PyTorch and when is it useful?

easy
  • A callback registered on a module that fires during the forward pass, receiving (module, input, output).
  • Useful for extracting intermediate features (e.g., pen-ultimate embeddings for downstream models), debugging shape / value issues, visualizing activations, running feature-map probes, or computing custom regularizers on hidden activations without modifying the model code.
  • Backward hooks exist too — for inspecting gradients per module.
Permalink

In one sentence, what is broadcasting and why does it matter?

easy
  • Broadcasting extends tensor operations across dimensions of unequal shape by implicit repetition, with rules: align shapes from the right, dimensions of size 1 are broadcast to match, otherwise dimensions must match.
  • It lets you write add, multiply, and mask operations across batches and features without explicit tile/repeat — critical for efficient GPU code.
Permalink

Compute the output shape of a 2D convolution with input (H, W), kernel k, stride s, padding p, dilation d.

medium
  • Hout  =  floorH_{\mathrm{out}}\; = \;\mathrm{floor}((H + 2*p - d*(k-1) - 1) / s) + 1, same for W.
  • Special cases: 'same' padding for stride 1 uses p = (k-1)/2 * d.
  • Standard kernels are 3×3 with p=1, s=1 (preserves spatial size).
  • Downsampling: stride 2 halves the resolution.
  • Understanding this formula is a bread-and-butter interview requirement — several bug fixes in production code hinge on it.
Permalink

Write the vanilla RNN update equation and explain why it struggles with long sequences.

easy
  • ht  =  tanh(Whh  ⋅  ht−1  +  Wxh  ⋅  xt  +  b)h_{t}\; = \;\mathrm{tanh}(W_{\mathrm{hh}}\; \cdot \;h_{t - 1}\; + \;W_{\mathrm{xh}}\; \cdot \;x_{t}\; + \;b).
  • Repeatedly multiplying by WhhW_{\mathrm{hh}} during BPTT causes gradients to vanish (spectral norm < 1) or explode (> 1) over long sequences.
  • LSTMs / GRUs introduce gated additive updates to preserve gradients across time; transformers skip recurrence entirely and use attention.
Permalink

MSE vs MAE for regression — when is each preferred?

easy
  • MSE (L2): differentiable everywhere, penalizes large errors quadratically → sensitive to outliers, gradients are proportional to error size.
  • MAE (L1): penalizes linearly → robust to outliers but not differentiable at 0.
  • Choose MSE when errors are Gaussian-ish and outliers are rare; choose MAE when the target has heavy tails or you want the median rather than the mean.
  • Huber loss is the practical compromise.
Permalink

Why is 'overfit a single batch' the first test you should run on a new model?

easy
  • Because it isolates implementation bugs from learning problems.
  • A correctly wired model with enough capacity can drive the loss on a handful of examples to nearly zero, since it can simply memorize them.
  • If it cannot, the fault is mechanical rather than statistical: labels misaligned with inputs, the loss reading the wrong axis, gradients not flowing because of a detached tensor or a frozen parameter, or a learning rate so small nothing moves.
  • The test costs seconds and rules out an entire class of silent failures before you spend GPU hours.
  • Only once a single batch overfits does it make sense to talk about regularization, augmentation, or architecture.
Permalink

The same training script gives different results on two runs. How much of that can you remove?

medium
  • Less than people expect, and it is worth knowing which sources you control.
  • Seeding the framework, Python and NumPy fixes weight initialization, dropout masks and shuffling order, which removes most of the variance.
  • What remains comes from the hardware: several GPU kernels, notably atomics used in scatter operations and some convolution algorithms, accumulate in nondeterministic order, and floating-point addition is not associative, so results differ in the last bits and then diverge over thousands of steps.
  • Frameworks expose a deterministic mode that swaps in slower ordered kernels, which gets you bit-identical runs at a real throughput cost.
  • Data loader worker count and any non-seeded augmentation also matter.
  • The practical stance is to seed everything, report results over several seeds rather than one, and reserve full determinism for debugging.
Permalink

What do residual connections actually fix?

medium
  • They make optimization tractable at depth.
  • Without them, a deep stack must learn an identity mapping through many nonlinear layers just to preserve information, which gradient descent does badly, and the observed symptom is that a deeper network trains to a worse training error than a shallower one.
  • That is an optimization failure, not overfitting.
  • A residual branch gives the gradient a path that reaches early layers with its magnitude roughly intact, so signal neither vanishes nor is distorted by the product of many Jacobians.
  • The broader consequence is that the network can represent a shallow function easily and add depth only where it helps, which is why residual blocks are in essentially every modern architecture.
Permalink
02

Backpropagation & autograd

2 concepts

Why does the backward pass require more memory than the forward pass?

medium
  • Reverse-mode autodiff needs the intermediate activations from the forward pass to compute gradients (the chain rule uses them).
  • So all activations are stored until backward finishes.
  • Peak memory ≈ activations + gradients + optimizer state.
  • Fixes for memory pressure: gradient checkpointing (recompute activations during backward instead of storing), mixed precision, model / ZeRO parallelism.
Permalink

What is backpropagation through time (BPTT)?

easy
  • Unroll the RNN over the sequence length T, treat it as a very deep feed-forward network with shared weights, and apply standard backprop.
  • Memory scales linearly with T.
  • Truncated BPTT limits the unroll length to k steps to bound memory (each backward pass covers only the most recent k timesteps), at the cost of not learning dependencies longer than k.
Permalink
03

Initialization & gradients

6 concepts

What are vanishing and exploding gradients, and how do you fix them?

medium
  • In deep networks, chain-rule products can shrink toward zero (vanishing) or blow up (exploding), stalling learning.
  • Fixes: better initializations (He for ReLU, Xavier for tanh), normalization (BatchNorm, LayerNorm), skip connections (ResNet), non-saturating activations (ReLU family), gradient clipping (for RNNs and large models), and appropriate learning rates.
Permalink

What is Xavier (Glorot) initialization and why?

medium
  • Sample weights from a distribution with variance 2  /  (fanin  +  fanout)2\; / \;(\mathrm{fan}_{\mathrm{in}}\; + \;\mathrm{fan}_{\mathrm{out}}) — designed so that activations and gradients have roughly the same variance layer to layer for tanh/sigmoid activations.
  • Prevents both vanishing and exploding signals at initialization.
  • Default in older frameworks; replaced by He init when ReLU became the standard.
Permalink

Why does He initialization use Var⁡  =  2  /  fanin\operatorname{Var}\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}} instead of Xavier's 2  /  (fanin  +  fanout)2\; / \;(\mathrm{fan}_{\mathrm{in}}\; + \;\mathrm{fan}_{\mathrm{out}})?

medium
  • ReLU zeros out half of its inputs on average, halving the effective variance of activations.
  • He init doubles the variance to compensate: Var⁡(W)  =  2  /  fanin\operatorname{Var}(W)\; = \;2\; / \;\mathrm{fan}_{\mathrm{in}}.
  • This keeps activation variance stable through ReLU layers and prevents signals from collapsing to zero.
  • Default for any modern ReLU / Leaky-ReLU / GELU network.
Permalink

When is orthogonal weight initialization useful?

hard
  • Orthogonal init sets weight matrices to random orthogonal matrices (singular values all 1), which preserves the norm of the input under linear map.
  • Especially useful for very deep networks and RNNs where repeated multiplication amplifies or shrinks signals — orthogonal keeps things unit-norm.
  • Slightly better than He on very deep nets without normalization.
Permalink

What causes exploding gradients and how do you handle them?

medium
  • Exploding gradients happen when the product of many derivatives with singular values > 1 grows exponentially.
  • Common in RNNs (repeated  multiplication  by  Whh)(\mathrm{repeated}\;\mathrm{multiplication}\;\mathrm{by}\;W_{\mathrm{hh}}) and very deep networks without normalization.
  • Signs: loss becomes NaN, weight norms blow up.
  • Fixes: gradient clipping (by  norm,  typical  max⁡norm  =  1−5)(\mathrm{by}\;\mathrm{norm}, \;\mathrm{typical}\;\operatorname{max}_{\mathrm{norm}}\; = \;1 - 5), spectral normalization, better initialization, residual connections, and lower learning rate.
Permalink

Gradient clipping by norm vs by value — which do you prefer?

easy
  • Clip-by-norm: rescale the whole gradient vector so its L2 norm is at most max⁡norm\operatorname{max}_{\mathrm{norm}}.
  • Preserves direction, only shrinks magnitude.
  • Preferred default.
  • Clip-by-value: clamp each element to [-clip, clip].
  • Distorts the gradient direction — element-wise clipping can flip the relative magnitudes of components.
  • Use clip-by-norm for RNNs and large-model training; clip-by-value only for specific numerical stability issues.
Permalink
04

Training dynamics

43 concepts

What does Batch Normalization do?

medium
  • For each mini-batch, BN normalizes activations to zero mean and unit variance per feature, then applies learned scale and shift.
  • Effects: faster convergence, higher learning rates, mild regularization, and reduced internal covariate shift.
  • Downsides: depends on batch size, awkward for small batches, and behaves differently in train vs eval mode.
  • LayerNorm (per-example) is preferred in transformers and RNNs.
Permalink

How does dropout work and when is it applied?

easy
  • During training, dropout randomly zeros a fraction p of activations per forward pass, forcing the network to distribute knowledge and not co-adapt.
  • At inference, all units are active and outputs are scaled (or scaling is done at train time — 'inverted dropout').
  • Common rates: 0.1-0.5.
  • Modern transformers use dropout in attention and MLP layers.
Permalink

Why use a learning-rate schedule (warmup + cosine decay)?

medium
  • Large models with LayerNorm behave badly at high LR early on — warmup ramps LR from 0 to peak over a few thousand steps to avoid divergence.
  • Then cosine (or linear) decay reduces LR smoothly toward zero, which improves final generalization.
  • Standard recipe for training transformers and LLMs.
Permalink

Why do residual (skip) connections help training deep networks?

easy
  • They provide a shortcut so gradients can flow directly back to earlier layers, alleviating vanishing gradients.
  • They also make it easy for a layer to learn the identity function, so adding depth can't hurt.
  • Residual blocks let us train 100+ layer networks (ResNet) and are used in transformers too (skip around attention and MLP).
Permalink

What is gradient accumulation and when do you use it?

easy
  • Run several forward-backward passes with small mini-batches without calling optimizer.step(); sum the gradients.
  • Then apply one optimizer step as if you had used a large batch.
  • Simulates a large effective batch on limited GPU memory without changing statistics much (BN stats still per-microbatch — so combine with GroupNorm or SyncBN if BN is used).
Permalink

How does batch size affect training dynamics?

medium
  • Larger batch → lower gradient noise, closer to true gradient, allows higher learning rate (roughly LR ∝ sqrt(batch)).
  • But too large batches tend to converge to sharper minima that generalize slightly worse (the 'generalization gap').
  • Small batches are noisier — regularizing effect that often helps final accuracy.
  • Rule: scale LR with batch and use warmup to keep training stable at very large batches (LARS/LAMB for extreme scales).
Permalink

What is deep double descent?

hard
  • Test error as a function of model capacity is not monotonically U-shaped: it goes up as capacity crosses the interpolation threshold (train error → 0), then goes down again for extremely overparameterized models.
  • Documented in Belkin et al. (2019) and Nakkiran et al. (2020).
  • Implication: in the overparameterized regime, bigger models can generalize better despite fitting the training data perfectly.
Permalink

What is grokking in deep learning?

hard
  • Phenomenon (Power et al., 2022) where a network trained on a small algorithmic task memorizes the training set quickly (train loss → 0) but validation stays random for a long time — then, after many more epochs of continued training, suddenly generalizes to near-perfect validation accuracy.
  • Suggests the network first memorizes, then slowly discovers a more compressed, generalizable circuit under continued regularization pressure.
Permalink

In one sentence, what is the Lottery Ticket Hypothesis?

hard
  • Frankle & Carbin (2019): a randomly initialized dense network contains sparse subnetworks ('winning tickets') that — when trained in isolation from their original initialization — reach comparable accuracy in comparable or fewer iterations.
  • Motivates pruning research: identify and train these tickets to shrink models dramatically without loss.
Permalink

What is the 'implicit bias' of SGD?

hard
  • Among the infinite minimizers of the training loss in an over-parameterized net, SGD tends to select ones with certain 'flat' properties — approximately minimum-norm solutions in linear cases, and empirically flat-minimum / max-margin solutions in nonlinear ones.
  • This implicit bias explains why deep nets trained by SGD generalize despite having capacity to fit noise.
  • Weight decay and small LR reinforce this bias.
Permalink

Your training loss becomes NaN. Debug it.

medium
  • (1) Check for divisions by zero and log of non-positive values — most common cause.
  • (2) Reduce learning rate; loss NaN often means gradient exploded.
  • (3) Add gradient clipping.
  • (4) Verify data has no NaN / inf.
  • (5) In fp16 / mixed precision, check the loss scale; use bf16 if available (more forgiving range).
  • (6) Reset from a known-good checkpoint.
  • (7) Print the loss and gradient norms every N steps in a rerun to isolate the offending step.
Permalink

What problem does LAMB solve?

hard
  • LAMB (Layer-wise Adaptive Moments for Batch training) enables very large-batch training (32k+ per step) without loss of accuracy.
  • It scales the Adam update per layer by the ratio ||θlayer{\theta}_{\mathrm{layer}}|| / ||updatelayer\mathrm{update}_{\mathrm{layer}}||.
  • Used to train BERT-Large in 76 minutes on TPU pods.
  • Similar spirit to LARS (which does the same for SGD).
Permalink

Describe the LR-range finder (Smith, 2015).

easy
  • Train the model for a few epochs with the learning rate increased exponentially from very small to very large.
  • Plot loss vs LR.
  • Pick a LR one order of magnitude below where the loss starts to diverge (or where the negative slope is steepest).
  • Fast heuristic — replaces manual grid search over the most important hyperparameter.
  • Available in fastai and PyTorch Lightning.
Permalink

What is the one-cycle policy?

medium
  • Smith's 'super-convergence' recipe: linearly ramp LR up from LRmax/25\mathrm{LR}_{\mathrm{max}} / 25 to LRmax\mathrm{LR}_{\mathrm{max}} over the first ~30% of training, then linearly decay back down.
  • Simultaneously modulate momentum in the opposite direction (high when LR is low, low when LR is high).
  • Enables very fast training with large max LR — regularizes via the 'LR-as-annealing-temperature' effect.
Permalink

How should learning rate scale with batch size?

medium
  • Linear rule (Goyal et al., 2017): LR  =  baseLR  ⋅  (batch  /  basebatch)\mathrm{LR}\; = \;\mathrm{base}_{\mathrm{LR}}\; \cdot \;(\mathrm{batch}\; / \;\mathrm{base}_{\mathrm{batch}}).
  • Works up to ~8k batch in vision with sufficient warmup.
  • Sqrt rule: LR ∝ sqrt(batch) — more conservative, sometimes better for Adam / transformers.
  • Very large batches (>32k) need layer-wise scaling (LARS, LAMB).
  • Always combine with warmup: large batches need longer warmup to avoid divergence.
Permalink

What is gradient noise scale (GNS) and how do you use it?

hard
  • GNS (McCandlish et al., 2018) measures the ratio between the variance of the mini-batch gradient and its squared mean, roughly telling you how much batch size you can grow before returns diminish.
  • Compute Bcrit  =  trace(Cov⁡(g))B_{\mathrm{crit}}\; = \;\mathrm{trace}(\operatorname{Cov}(g)) / ||g||².
  • Below BcritB_{\mathrm{crit}}, doubling batch halves steps.
  • Above, benefits plateau.
  • Guides choice of batch size for large-model training.
Permalink

What is LARS and when is it used?

hard
  • Layer-wise Adaptive Rate Scaling (You et al., 2017).
  • Scales the update per layer by ||θlayer{\theta}_{\mathrm{layer}}|| / ||glayerg_{\mathrm{layer}}||, keeping the ratio update-to-weight at ~1%.
  • Enables training ResNet-50 in a few minutes with batch sizes up to 32k.
  • Same idea as LAMB but for SGD-with-momentum.
  • Standard for very-large-batch training in vision.
Permalink

Why does a model with BatchNorm behave differently at train time vs eval time?

medium
  • Train: BN uses batch statistics (mean/var of the current mini-batch), providing regularization noise.
  • Eval: uses stored running averages accumulated during training via EMA.
  • If those running stats are out of sync with training (small batches, non-i.i.d. batches, incorrect running momentum), eval accuracy can drop dramatically.
  • Common bug: fine-tuning with BN in train mode but small batches contaminates running stats.
Permalink

What is stochastic depth / DropPath and where is it used?

medium
  • During training, randomly drop entire residual branches (set them to 0 and pass identity through the skip connection) with some probability p that increases with depth.
  • Effectively trains an ensemble of networks of varying depth.
  • Standard trick in modern deep vision transformers (Swin, ConvNeXt, DeiT) — enables much deeper training at higher accuracy.
Permalink

What is early stopping and how do you configure it?

easy
  • Monitor a validation metric each epoch; stop training when it hasn't improved for 'patience' epochs.
  • Restore the best-performing checkpoint.
  • Prevents overfitting when training beyond the optimal point.
  • Config: choose the right metric (accuracy for classification, F1 for imbalanced, loss for regression), patience ~5-20% of total epochs, and a small delta threshold to avoid stopping on noise.
Permalink

What is Stochastic Weight Averaging (SWA)?

medium
  • During the last part of training (with a constant or cyclical LR), keep a running average of the weights: θswa  =  mean{\theta}_{\mathrm{swa}}\; = \;\mathrm{mean} of θt{\theta}_{t} across recent steps.
  • Use θswa{\theta}_{\mathrm{swa}} at inference.
  • Averaging in the loss landscape moves toward the center of a flat minimum, which generalizes better than any individual point.
  • Adds ~0 cost and often gains 0.5-1% accuracy.
  • Extended to EMA of weights (used in EMA-teacher / EMA distillation).
Permalink

How does snapshot ensembling work?

medium
  • Train with a cyclical LR schedule (SGDR / cosine with restarts).
  • Save a checkpoint at each cycle's minimum LR (one snapshot per cycle).
  • At inference, ensemble the K snapshots by averaging predictions.
  • Same training cost as a single training run, but ensemble diversity comes from the LR restarts landing in different basins.
Permalink

What is adversarial training (PGD, FGSM)?

hard
  • Generate small adversarial perturbations that fool the current model (FGSM = one gradient step; PGD = multiple projected gradient steps), then include them in training with the true labels.
  • Improves robustness to bounded input perturbations at ~2-10x compute overhead.
  • Trade-off: clean accuracy typically drops 3-5% while adversarial accuracy jumps from ~0% to 40-60%.
  • Standard baseline for robust ML.
Permalink

How are input images typically normalized for pretrained CNNs?

easy
  • Subtract ImageNet channel means [0.485, 0.456, 0.406] and divide by stds [0.229, 0.224, 0.225] (RGB).
  • This matches the distribution the model saw during pretraining — using different stats can hurt accuracy noticeably during transfer.
  • When training from scratch, either compute stats on your dataset or normalize to [-1, 1] / [0, 1].
  • Store the norm inside the model's preprocess step so it can't be forgotten in production.
Permalink

What is a solid default augmentation recipe for training a modern vision model?

medium
  • Random resized crop, horizontal flip, color jitter (brightness/contrast/saturation/hue), and one of RandAugment / TrivialAugment for stronger transforms.
  • Add Random Erasing / Cutout.
  • For classification, layer in Mixup (α=0.2) and CutMix (α=1.0).
  • Label smoothing 0.1.
  • Combined with AdamW + cosine schedule + EMA, this recipe gives you ~+2-4% over vanilla training on ImageNet and is the modern baseline (DeiT / ConvNeXt).
Permalink

How should training resolution be chosen for a CNN or ViT?

medium
  • For CNNs: bigger resolution → higher accuracy up to a point (diminishing returns and OOM).
  • Common choices: 224 for ResNet baselines, 256/288/380/456 for EfficientNet variants (compound scaling), 384/512 for ViT-Large.
  • Train at moderate resolution, then fine-tune at higher resolution ('progressive resizing') for a cheap accuracy boost.
  • For detection / segmentation, resolution matters more — small objects need high input res.
Permalink

When do you use truncated BPTT and what's the trade-off?

medium
  • For very long sequences (language modeling on 1000+ tokens, time-series, audio).
  • Unroll for k steps, backward, then carry hidden state forward but detach it from the graph.
  • Bounded memory, faster training.
  • Trade-off: can't learn dependencies longer than k without further tricks (e.g., overlap windows, memorized state).
  • Modern transformers avoid this via full parallel attention over the whole context.
Permalink

What is teacher forcing and its main pitfall?

medium
  • During training, feed the ground-truth previous token as input to the decoder at each step (instead of the model's own previous prediction).
  • Speeds training and avoids compounding errors from incorrect early predictions.
  • Pitfall: exposure bias — inference behavior differs from training (model has never seen its own errors).
  • Fixes: scheduled sampling (mix true vs predicted with a schedule) or reinforcement fine-tuning.
Permalink

What is scheduled sampling?

medium
  • Interpolate between teacher forcing and self-generation during training: with probability p feed the true token, with 1-p feed the model's own prediction.
  • Decay p over training (start at 1, end near 0).
  • Bridges the train/inference gap without full exposure.
  • Works OK for RNN seq2seq but doesn't compose easily with parallel transformer training (which processes all positions at once).
Permalink

Contrast the pretraining objectives of BERT, GPT, and T5.

medium
  • BERT: masked language modeling — mask 15% of tokens, predict them (bidirectional context).
  • GPT: causal / next-token prediction — predict token t from tokens 1..t-1 (unidirectional).
  • T5: 'span corruption' — replace random spans with sentinel tokens, decoder generates the missing spans (encoder-decoder).
  • Each objective aligns with the target usage: BERT for embeddings, GPT for generation, T5 for text-to-text.
Permalink

What is deep supervision / auxiliary loss?

medium
  • Attach small classification / regression heads to intermediate layers and add their losses to the main loss (with lower weights).
  • Provides gradient signal deeper into the network — helps optimization of very deep nets and enables intermediate feature usefulness.
  • Used in Inception (auxiliary classifiers), U-Net (deep supervision on multi-scale masks), and dense prediction.
  • Modern residual nets need it less.
Permalink

How does mixed-precision training work with fp16 / bf16?

medium
  • Store weights in fp32, cast to fp16 (or bf16) for forward and backward passes, accumulate gradients in fp32, then update weights in fp32.
  • Cuts activation memory ~2x and speeds up compute 2-3x on modern GPUs with tensor cores. fp16 has small range (needs loss scaling to prevent gradient underflow); bf16 has fp32's range but less precision — usually plug-and-play, preferred on Ampere+.
  • Use torch.cuda.amp / bf16 autocast.
Permalink

Why does fp16 training need loss scaling?

hard
  • fp16 has a small dynamic range (~5e-8 to 6.5e4).
  • Gradients often live near the underflow boundary — many gradient elements silently become zero.
  • Solution: multiply loss by a scale factor S (e.g., 128) before backward → gradients are S× larger → survive fp16 range → divide by S before optimizer step.
  • Dynamic loss scaling adjusts S automatically: increase when no NaN, halve when a NaN appears.
  • Not needed with bf16 (has fp32's exponent range).
Permalink

How does PyTorch DDP work under the hood?

medium
  • Each GPU has a full replica of the model + its own data slice.
  • Forward and backward happen independently.
  • When a bucket of gradients is computed (during backward), DDP does an all-reduce across all GPUs so every rank ends up with the averaged gradient.
  • Optimizer.step is then identical across GPUs → weights stay in sync.
  • Gradient bucketing overlaps communication with computation, hiding all-reduce latency.
Permalink

Model parallel vs data parallel — when do you need each?

hard
  • Data parallel: replicate the model, split the batch across GPUs — dominant approach up to ~10B params on a single node.
  • Model parallel: split the model itself across GPUs — needed when the model exceeds one GPU's memory.
  • Two flavors: tensor parallel (split each matrix multiply across GPUs, e.g., Megatron) and pipeline parallel (put different layers on different GPUs, with microbatching to keep them all busy).
  • Modern LLM training combines all three (3D parallelism).
Permalink

Explain the three ZeRO stages.

hard
  • ZeRO (Zero Redundancy Optimizer) partitions training state across GPUs to save memory.
  • Stage 1: shard the optimizer state (Adam's m and v) across N GPUs → memory / N.
  • Stage 2: also shard the gradients.
  • Stage 3 (FSDP): also shard the parameters themselves — each GPU only holds a 1/N slice of the model at rest and gathers full weights only when needed.
  • Enables training LLMs many times bigger than a single-GPU capacity.
Permalink

How is FSDP different from DDP?

hard
  • PyTorch FSDP (Fully Sharded Data Parallel) implements ZeRO-3: each GPU stores only a shard of the parameters, gradients, and optimizer state.
  • When a layer runs forward, FSDP all-gathers its full parameters, computes, then discards them (or shards again).
  • Enables 10-100x larger models than DDP on the same hardware, at the cost of extra communication.
  • Modern default for training large models in PyTorch.
Permalink

What is an EMA of weights and why do modern training recipes use it?

medium
  • Maintain a shadow copy of the weights θema{\theta}_{\mathrm{ema}} updated as θema{\theta}_{\mathrm{ema}} ← α  ⋅  θema  +  (1−α)  ⋅  θ{\alpha}\; \cdot \;{\theta}_{\mathrm{ema}}\; + \;(1 - {\alpha})\; \cdot \;{\theta} every step, with α ~ 0.999-0.9999.
  • Evaluate / deploy with θema{\theta}_{\mathrm{ema}}, not the raw training weights.
  • The EMA lives at the center of a flat minimum → ~0.5-1% better accuracy for free, better calibration, more stable predictions.
  • Widely used in diffusion training (essential for sample quality), BYOL / DINO self-supervised methods (EMA teacher), and modern classification recipes.
Permalink

State the Chinchilla scaling insight in one sentence.

hard
  • For a given compute budget C = 6 * N * T (params × tokens), the compute-optimal split has N and T roughly proportional (~20 tokens per parameter), meaning older large models (GPT-3, Gopher) were undertrained: same compute would have given a smaller model trained on more data with much better loss.
  • Reshaped modern LLM training toward smaller-but-longer-trained models (LLaMA, Mistral).
Permalink

Your validation loss is lower than your training loss. Is something broken?

medium
  • Usually not, and there are three ordinary explanations before you go looking for a bug.
  • Regularization is active during training but not at evaluation, so dropout and stochastic depth make the training forward pass genuinely harder than the validation one.
  • Augmentation does the same thing more strongly, since the model is scored on clean images but trained on distorted ones.
  • And the reported training loss is an average over the epoch while the model was still improving, whereas validation is measured once at the end, so the training number reflects an older, worse model by roughly half an epoch.
  • The signs that it really is a bug are a validation set that is easier than the training set because of a bad split, leakage of training examples into validation, or evaluation code that silently skips the hard cases.
Permalink

Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?

hard
  • Batch norm estimates the mean and variance from the batch itself, so with four samples those statistics are extremely noisy.
  • The noise acts as an uncontrolled regularizer during training and, worse, the running averages accumulated for inference no longer match what the layer saw, so train and eval behaviour diverge.
  • It also couples examples within a batch, which breaks any assumption of per-sample independence.
  • The practical fixes are to switch to group norm or layer norm, whose statistics are computed per sample and are therefore batch-size independent, or to use a normalization-free architecture with careful initialization.
  • This is exactly why detection and segmentation models, which run tiny batches of large images, standardly use group norm.
Permalink

Your augmentation pipeline made validation accuracy worse. What went wrong?

medium
  • Almost always a distribution mismatch: the augmentation created inputs unlike anything at test time, so the model spent capacity on a harder problem than the one it is graded on.
  • Classic examples are horizontal flips on digits or text, aggressive colour jitter when colour is the label signal, and rotations on medical scans with a canonical orientation.
  • Second possibility is that augmentation was accidentally applied to the validation set, which changes the measurement rather than the model.
  • Third, the augmentation is fine but too strong for the training budget, since heavier augmentation needs more epochs to pay off.
  • The diagnostic is to look at augmented samples with your own eyes and ask whether the label is still correct and the image still plausible.
Permalink

How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?

medium
  • You run several smaller micro-batches, summing or averaging their gradients, and only step the optimizer once at the end.
  • Mathematically the update matches a single large batch, so the loss curve is nearly identical while peak activation memory stays at the micro-batch level.
  • What it does not fix is time: you still do the same amount of computation, so the step takes proportionally longer, and it does not give the throughput benefit of a genuinely larger batch on more hardware.
  • It also interacts badly with batch normalization, whose statistics are computed per micro-batch and therefore reflect the small size, not the effective one.
  • Remember to scale the loss so accumulation averages rather than sums.
Permalink
05

Optimizers

17 concepts

SGD vs Adam vs AdamW — how do you choose?

medium
  • SGD with momentum is simple and often generalizes better on vision when tuned carefully with LR schedules.
  • Adam adapts per-parameter learning rates using first and second moment estimates — great default, converges fast, robust to hyperparameters.
  • AdamW decouples weight decay from the gradient update — the standard for training large transformers and modern LLMs. Rule of thumb: AdamW for transformers, SGD+momentum for classic CNNs.
Permalink

Write the vanilla SGD update rule and explain each term.

easy
  • θt+1  =  θt  −  η  ⋅  gt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;g_{t}, where θ is the parameter vector, η the learning rate, and gtg_{t} = ∇_θ L(θt  batcht)L({\theta}_{t}\;\mathrm{batch}_{t}) is the stochastic gradient on the current mini-batch.
  • Each step is a noisy estimate of the true gradient — the noise itself acts as regularizer and helps escape saddle points.
  • Convergence rate for convex problems: O(1/√T).
Permalink

How does classical momentum modify SGD?

easy
  • Maintain a velocity: vt  =  β  ⋅  vt−1  +  gtv_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;g_{t}; then θt+1  =  θt  −  η  ⋅  vt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;v_{t}.
  • Typical β = 0.9.
  • Momentum accumulates gradients over recent steps, damping oscillations along high-curvature directions and accelerating along consistent ones.
  • Roughly equivalent to averaging gradients over the last ~1/(1-β) steps — often improves convergence 2-10x on ill-conditioned problems.
Permalink

What is Nesterov momentum and how is it different from classical momentum?

medium
  • Look-ahead trick: compute the gradient at the parameters shifted by the current velocity, not at the current parameters.
  • Update: vt  =  β  ⋅  vt−1v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1 + ∇L(θt  −  η  ⋅  β  ⋅  vt−1)L({\theta}_{t}\; - \;{\eta}\; \cdot \;{\beta}\; \cdot \;v_{t} - 1); θt+1  =  θt  −  η  ⋅  vt{\theta}_{t} + 1\; = \;{\theta}_{t}\; - \;{\eta}\; \cdot \;v_{t}.
  • Effectively 'peek' where momentum would take you and correct.
  • Theoretical acceleration in convex settings (O(1/T2)  instead  of  O(1/T))(O(1 / T^{2})\;\mathrm{instead}\;\mathrm{of}\;O(1 / T)), modest improvement in deep learning practice.
Permalink

What is AdaGrad and its main drawback?

medium
  • Per-parameter learning rate: θ ← θ - η * g / (sqrt(sum of squared past gradients) + ε).
  • Parameters with large historical gradients get smaller effective LRs — great for sparse features (NLP with one-hot inputs).
  • Drawback: accumulator grows monotonically, so effective LR → 0.
  • Model stops learning after enough steps.
  • RMSProp and Adam fix this with exponential moving averages.
Permalink

How does RMSProp fix AdaGrad?

medium
  • Replaces the running sum of squared gradients with an exponential moving average: vt  =  β  ⋅  vt−1  +  (1−β)  ⋅  gt2v_{t}\; = \;{\beta}\; \cdot \;v_{t} - 1\; + \;(1 - {\beta})\; \cdot \;g_{t}^{2}.
  • Update: θ ← θ  −  η  ⋅  g  /  (sqrt(vt)  +  ε){\theta}\; - \;{\eta}\; \cdot \;g\; / \;(\mathrm{sqrt}(v_{t})\; + \;{\varepsilon}).
  • Typical β = 0.9-0.99.
  • LR no longer decays to zero — the running variance forgets old gradients.
  • Good default for RNNs before Adam took over.
Permalink

Write Adam's full update rule.

medium
  • mt  =  β1  ⋅  mt−1  +  (1−β1)  ⋅  gtm_{t}\; = \;{\beta}_{1}\; \cdot \;m_{t} - 1\; + \;(1 - {\beta}_{1})\; \cdot \;g_{t} (first moment); vt  =  β2  ⋅  vt−1  +  (1−β2)  ⋅  gt2v_{t}\; = \;{\beta}_{2}\; \cdot \;v_{t} - 1\; + \;(1 - {\beta}_{2})\; \cdot \;g_{t}^{2} (second moment).
  • Bias-corrected: mhat  =  mt  /  (1  −  β1t)m_{\mathrm{hat}}\; = \;m_{t}\; / \;(1\; - \;{\beta}_{1}^{t}), vhat  =  vt  /  (1  −  β2t)v_{\mathrm{hat}}\; = \;v_{t}\; / \;(1\; - \;{\beta}_{2}^{t}).
  • Update: θ ← θ  −  η  ⋅  mhat  /  (sqrt(vhat)  +  ε){\theta}\; - \;{\eta}\; \cdot \;m_{\mathrm{hat}}\; / \;(\mathrm{sqrt}(v_{\mathrm{hat}})\; + \;{\varepsilon}).
  • Defaults: β1=0.9{\beta}_{1} = 0.9, β2=0.999{\beta}_{2} = 0.999, ε=1e-8.
  • Combines momentum and per-param LR — robust default optimizer.
Permalink

Why does AdamW work better than Adam + L2 weight decay for transformers?

hard
  • In Adam, adding λ*θ inside the gradient means weight decay gets rescaled by the per-parameter learning rate 1/sqrt(vhat)1 / \mathrm{sqrt}(v_{\mathrm{hat}}).
  • Parameters with small gradients get almost no decay; those with large ones get too much.
  • AdamW decouples: apply weight decay directly to parameters (θ ← θ  −  η⋅(mhat/sqrt(vhat)  +  λ⋅θ){\theta}\; - \;{\eta} \cdot (m_{\mathrm{hat}} / \mathrm{sqrt}(v_{\mathrm{hat}})\; + \;{\lambda} \cdot {\theta})) — every parameter shrinks by η*λ uniformly.
  • Critical for transformer training.
Permalink

What is the Lion optimizer and why is it interesting?

hard
  • Lion (Chen et al., 2023, from Google) uses only the sign of the momentum: θ ← θ  −  η  ⋅  sign⁡(β1⋅m  +  (1−β1)⋅g){\theta}\; - \;{\eta}\; \cdot \;\operatorname{sign}({\beta}_{1} \cdot m\; + \;(1 - {\beta}_{1}) \cdot g), then m ← β2⋅m  +  (1−β2)⋅g{\beta}_{2} \cdot m\; + \;(1 - {\beta}_{2}) \cdot g.
  • Half the optimizer state of Adam (no v), often trains 1.5-2x faster in wall-clock at same accuracy on transformers and vision.
  • Requires smaller LR (~10x smaller than Adam) and more aggressive weight decay.
Permalink

Why does Adafactor use less memory than Adam?

hard
  • Adafactor factorizes the second-moment matrix v: instead of storing a full vtv_{t} per parameter (O(N) memory), it stores row-sum and column-sum statistics for each 2D weight matrix (O(sqrt(N))).
  • Massive memory savings — enabled T5 training.
  • Trade-off: some hyperparameters harder to tune, occasionally less stable than full Adam.
Permalink

What is Shampoo and when does second-order optimization pay off?

hard
  • Shampoo (Gupta et al., 2018 / Anil et al., 2020) is a second-order-ish optimizer that maintains a per-parameter matrix preconditioner factorized along tensor axes — computes the inverse Kronecker-factored covariance of gradients.
  • Uses more compute per step than Adam but converges in far fewer steps.
  • Practical for very large-scale training where compute is limited by memory bandwidth rather than FLOPs (T5, PaLM experiments).
Permalink

How does weight decay change the objective and the update?

medium
  • Adds an L2 penalty λ/2 * ||θ||² to the loss.
  • In SGD: θ ← θ - η * (∇L + λ*θ) = (1 - η*λ) * θ - η * ∇L — shrinks weights each step.
  • Effects: prevents overfitting (Occam-style bias toward small weights), reduces effective model capacity, and improves generalization.
  • In AdamW, applied directly to θ (not through g), which is why AdamW works better than 'Adam + L2 in loss'.
Permalink

How much extra memory does Adam use versus SGD?

medium
  • SGD with momentum: 1x parameter count in optimizer state (momentum).
  • Adam: 2x parameter count (first and second moment estimates).
  • AdamW: same as Adam.
  • For a 7B model in fp32, this is 28 GB just for m and v — a big reason why large-model training uses fp16 / bf16 optimizer states, sharded optimizers (ZeRO), or memory-efficient optimizers (Adafactor, 8-bit Adam).
Permalink

What does SAM (Sharpness-Aware Minimization) do?

hard
  • SAM (Foret et al., 2020) minimizes a surrogate that penalizes sharp minima: for each step, first perturb θ in the direction that maximizes the loss within a small ball (θ  +  ρ  ⋅  ∇L  /  ∣∣∇L∣∣)({\theta}\; + \;{\rho}\; \cdot \; \nabla L\; / \; \mid \mid \nabla L \mid \mid ), then compute the gradient at that perturbed point and update the original θ.
  • Roughly 2x compute per step, but consistently improves generalization on vision and NLP benchmarks.
  • Extended in adaptive SAM variants.
Permalink

Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?

hard
  • L2 in loss: gradient becomes g + λ*θ, then Adam divides by sqrt(vhat)\mathrm{sqrt}(v_{\mathrm{hat}}).
  • Parameters with large gradients get smaller effective weight decay — inverted relative to what you want.
  • Decoupled (AdamW): weight decay applied directly to θ as θ ← θ - η*λ*θ, independent of the adaptive scale.
  • Every parameter shrinks uniformly.
  • Empirically much better generalization on transformers.
Permalink

When do you prefer SGD with momentum over Adam?

medium
  • For convolutional vision models trained from scratch to their best possible accuracy, tuned SGD with momentum still tends to generalize slightly better than Adam, which is why many image classification recipes use it.
  • Adam wins where gradient scales differ wildly across parameters: transformers, embeddings with sparse updates, and anything with attention.
  • Adam is also far more forgiving of a poorly chosen learning rate, which makes it the right default when you cannot afford a sweep.
  • Memory matters too, since Adam stores two extra states per parameter, which is significant at scale.
  • The honest summary is that Adam, or AdamW, is the default for language models and SGD remains competitive for vision, and the difference shrinks once both are properly tuned.
Permalink

Why does AdamW handle weight decay differently from Adam, and why does it matter?

hard
  • In Adam, weight decay is added into the gradient, so it passes through the adaptive scaling.
  • Parameters with a large accumulated second moment get their decay divided down, which means the effective regularization differs per parameter and does not match what you asked for.
  • AdamW decouples it: the decay is applied directly to the weights, outside the adaptive step, so every parameter shrinks at the same relative rate.
  • In practice this makes the weight decay hyperparameter behave predictably and transfer between learning rates, and it measurably improves generalization on transformers.
  • It is also why decay is usually excluded from bias and normalization parameters, where shrinking towards zero has no regularizing meaning.
Permalink
06

Learning-rate schedules

7 concepts

What are cyclical learning rates (CLR)?

medium
  • Instead of monotonic decay, oscillate LR between a low and a high bound in periodic triangular / sinusoidal / exp cycles.
  • Helps escape shallow local minima (high LR periods) and refines fits (low LR periods).
  • Underlying insight: SGD is a stochastic process — periodic 'heating' explores basins, 'cooling' exploits them.
  • Precursor to SGDR / cosine with restarts.
Permalink

Why do transformers need learning-rate warmup?

medium
  • At the start of training, weights are random and LayerNorm statistics are meaningless.
  • A high LR causes large gradients that destabilize LayerNorm's running scale and can permanently damage the network (attention heads collapse).
  • Warmup ramps LR linearly from 0 to peak over a few thousand steps, giving the network time to stabilize activations before high-LR updates.
Permalink

What is cosine annealing and why is it preferred over step decay?

medium
  • LR(t) = LR_min + 0.5 * (LR_max - LR_min) * (1 + cos(π * t / T)).
  • Starts high, decays smoothly to LRmin\mathrm{LR}_{\mathrm{min}} over T steps.
  • Smoother than step decay (no sudden LR jumps that upset momentum), typically better final accuracy on transformers and CNNs.
  • Combined with warmup, it's the de-facto schedule for modern large-model training.
Permalink

When is step decay still appropriate?

easy
  • Multiply LR by a factor (e.g., 0.1) at fixed milestones (e.g., 60/90 epochs on ImageNet).
  • Simple, deterministic, well-understood — classic recipe for training ResNet-style CNNs with SGD+momentum.
  • Downsides: sudden LR jumps interact poorly with running momentum, need careful tuning of milestones.
  • Cosine is smoother but step decay is fine for well-studied recipes.
Permalink

How does ReduceLROnPlateau work?

easy
  • Track a validation metric; if it doesn't improve for 'patience' epochs, multiply LR by a factor (e.g., 0.5).
  • Adaptive to the actual loss curve — doesn't require guessing the schedule in advance.
  • Good default for prototyping.
  • Downside: introduces validation feedback into training, and its aggressiveness (patience, factor) matters a lot.
Permalink

What is SGDR (warm restarts) and when does it help?

medium
  • Cosine annealing with periodic restarts back to LRmax\mathrm{LR}_{\mathrm{max}}: each 'cycle' cools LR to LRmin\mathrm{LR}_{\mathrm{min}} then jumps back up.
  • Enables ensembling by saving one model per cycle (Snapshot Ensembles).
  • Helps escape sharp minima and revisits the loss surface.
  • Widely used in vision competitions and image-classification recipes; less common in transformers where a single cosine schedule dominates.
Permalink

Why does transformer training usually need a learning-rate warmup?

hard
  • At initialization, Adam's second-moment estimate is based on almost no history, so its normalization is unreliable and the first updates can be enormous relative to the weights.
  • Large early steps push a transformer into a region from which it never fully recovers, often visible as an attention collapse or an immediate loss spike.
  • Warmup ramps the learning rate from near zero over a few hundred to a few thousand steps, giving the optimizer state time to become meaningful before large steps are taken.
  • Post-layer-norm architectures need it most, because gradients through the residual path are badly scaled at initialization; pre-layer-norm and careful residual scaling reduce the dependence but rarely remove it entirely.
Permalink
07

Normalization

9 concepts

When do you use LayerNorm instead of BatchNorm?

medium
  • Use LayerNorm when batch size is small or varies (RNNs, transformers, sequence data).
  • LayerNorm computes statistics across features within a single example, so it doesn't depend on batch statistics.
  • That's why it's standard in transformers and language models where sequence lengths vary and batches can be tiny.
Permalink

What is GroupNorm and when do you use it?

medium
  • Divide channels into G groups, normalize each group per example.
  • No dependence on batch size — works with batch=1, useful for detection / segmentation where memory forces small batches, or for very high-resolution training.
  • Middle ground between LayerNorm (all channels one group) and InstanceNorm (each channel its own group).
  • Standard in Detectron2, MMDetection, and small-batch computer vision.
Permalink

What is InstanceNorm and where does it shine?

medium
  • Normalize each channel independently within one example.
  • Removes per-image style / contrast information — hence its use in style transfer and image-to-image translation (CycleGAN, StyleGAN).
  • The scale/shift then allow the network to inject any desired 'style'.
  • Not usually used for classification because it destroys instance-specific information that helps discrimination.
Permalink

What is Weight Normalization and its trade-off?

medium
  • Reparameterize weights as w = g * v / ||v||, decoupling magnitude (g, scalar per neuron) from direction (v, vector).
  • Data-independent normalization — no batch statistics.
  • Cheaper than BN, useful in RNNs and RL where BN struggles.
  • Trade-off: less regularization than BN and worse convergence in most modern CNNs and transformers, so it's rarely used today.
Permalink

How does RMSNorm differ from LayerNorm?

medium
  • RMSNorm(x) = x / RMS(x) * γ, where RMS(x)  =  sqrt(mean(x2)  +  ε)\mathrm{RMS}(x)\; = \;\mathrm{sqrt}(\mathrm{mean}(x^{2})\; + \;{\varepsilon}).
  • Skips the mean-subtraction and β (bias) of LayerNorm.
  • About 10-30% faster (fewer ops), similar accuracy on transformers.
  • Adopted by LLaMA, T5, and most 2023+ open LLMs.
Permalink

Pre-norm vs post-norm transformers — which is standard and why?

hard
  • Original Transformer (Vaswani 2017) used post-norm: x + Attn(x), then LayerNorm.
  • Trains poorly at scale (deep post-norm nets need long warmup).
  • Pre-norm applies LayerNorm before the sublayer: x + Attn(LN(x)) — gradients flow through the identity skip unchanged, trains stably with very deep nets and short warmup.
  • All modern LLMs use pre-norm.
Permalink

What is Synchronized BatchNorm (SyncBN) and when do you need it?

medium
  • In data-parallel training, each GPU has a mini-batch — regular BN computes stats per GPU.
  • If per-GPU batch is small (e.g., 2-4 for detection / segmentation), stats are noisy and accuracy suffers.
  • SyncBN aggregates mean and variance across all GPUs each forward pass, giving one global batch statistic.
  • Slower (extra collective), but essential for dense-prediction tasks with small per-GPU batch.
Permalink

Why is BatchNorm awkward in RNNs?

medium
  • RNNs share weights across time; you'd need separate BN statistics for every timestep to be principled, but sequences have variable length.
  • Also, small effective batch (batch × time reshape doesn't help because time is not exchangeable), and running stats depend on sequence length.
  • LayerNorm has none of these issues — per-example, per-timestep, unconditional on batch.
  • That's why LSTMs and transformers use LN, not BN.
Permalink

Why did transformers switch from post-norm to pre-norm at scale?

hard
  • Original transformer used post-norm: LayerNorm(x + sublayer(x)).
  • At depth > 12, the sublayer outputs interfere with LN's running scale, causing training instabilities that require very long warmup or careful initialization.
  • Pre-norm — x + sublayer(LayerNorm(x)) — puts the LayerNorm inside the residual branch, leaving a clean identity path for gradients.
  • Trains stably for 100+ layers with short warmup.
  • All modern LLMs use pre-norm.
Permalink
08

Regularization

9 concepts

What is DropConnect and how is it different from Dropout?

hard
  • Dropout zeros activations; DropConnect zeros weights (elements of the weight matrix) — i.e., randomly severs individual connections.
  • More granular regularization but harder to implement efficiently on GPUs.
  • Occasionally used in specialized architectures but Dropout dominates in practice due to speed.
Permalink

Where is dropout typically inserted in a transformer block?

medium
  • (1) On attention weights (after softmax), (2) on attention output projections, (3) on MLP hidden and output activations, (4) on residual sums (dropout after each residual add — 'residual dropout').
  • Rates: ~0.1 for pretraining (BERT), sometimes 0.0 for LLM pretraining (large models overfit less).
  • Fine-tuning usually adds dropout on the classifier head.
Permalink

What is label smoothing and why does it help?

easy
  • Replace one-hot target [0, ..., 1, ..., 0] with (1-α)*one-hot + α/K (uniform), typically α=0.1.
  • Prevents the network from becoming over-confident: it can't drive the softmax logit for the true class to infinity because the target isn't exactly 1.
  • Improves calibration, reduces overconfidence, and often improves accuracy 0.5-1% on classification benchmarks.
  • Standard in ImageNet training and NMT.
Permalink

How does Mixup work?

medium
  • Sample two examples (xi,  yi)(x_{i}, \;y_{i}), (xj,  yj)(x_{j}, \;y_{j}) and a mixing coefficient λ ~ Beta(α, α) with α ~ 0.2.
  • Train on the interpolation: xnew  =  λ⋅xi  +  (1−λ)⋅xjx_{\mathrm{new}}\; = \;{\lambda} \cdot x_{i}\; + \;(1 - {\lambda}) \cdot x_{j}, ynew  =  λ⋅yi  +  (1−λ)⋅yjy_{\mathrm{new}}\; = \;{\lambda} \cdot y_{i}\; + \;(1 - {\lambda}) \cdot y_{j}.
  • Forces linearity between examples — smoother decision boundaries, better calibration, robustness to adversarial noise.
  • Widely used in vision.
  • Doesn't work as well on text without variants.
Permalink

What is Cutout / Random Erasing?

easy
  • During training, mask a random rectangular region of the input image with zeros (Cutout) or random noise (Random Erasing).
  • Forces the network to look at the whole image rather than one salient region — improves robustness to occlusion.
  • Simple, cheap, effective.
  • Standard in vision recipes since 2017; combines well with Mixup / CutMix.
Permalink

Why is data augmentation effectively free regularization?

easy
  • Every augmentation is a prior about invariances the true function satisfies (translations of a cat are still a cat).
  • Applying augmentations enlarges the effective training distribution without labeling cost.
  • Forces the model to be invariant to those transformations, which reduces overfitting to specific instances.
  • Free lunch in vision; needs task-specific care in text and tabular.
Permalink

What is consistency regularization?

medium
  • Enforce that two augmented views of the same unlabeled input produce similar predictions: Lcons  =  DL_{\mathrm{cons}}\; = \;D(f(aug1(x)), f(aug2(x))) with D = KL or MSE.
  • Powers modern semi-supervised methods (FixMatch, Mean Teacher, MixMatch) and self-supervised learning (SimCLR, MoCo, BYOL).
  • Cornerstone technique when labeled data is scarce.
Permalink

What is Manifold Mixup?

hard
  • Variant of Mixup (Verma et al., 2019): at each step, pick a random hidden layer k and linearly interpolate the hidden representations of two examples at that layer (rather than at the input).
  • Encourages smoother hidden representations, further improves calibration and robustness.
  • Slightly more expensive per step, marginal gains over vanilla Mixup on top-tier vision benchmarks.
Permalink

Why does dropout play a smaller role in large language model training than in older vision models?

hard
  • Because the regularization pressure comes from the data instead.
  • When a model sees each token roughly once, on a corpus far larger than its parameter count, memorization is not the binding constraint, so the variance reduction dropout provides is not needed and its added gradient noise mostly slows convergence.
  • Older vision models made many passes over a small labelled set, where overfitting was the dominant failure and dropout genuinely helped.
  • Modern large-scale pretraining therefore uses little or no dropout, relying on data volume, weight decay and early stopping in the form of a single epoch.
  • Dropout returns during fine-tuning on small task datasets, where the classic overfitting regime is back.
Permalink
09

Data augmentation & mixing

3 concepts

How does CutMix differ from Mixup?

medium
  • Cut a rectangular patch from image B and paste it into image A.
  • The label is mixed by the area ratio: y  =  λ⋅yA  +  (1−λ)⋅yBy\; = \;{\lambda} \cdot y_{A}\; + \;(1 - {\lambda}) \cdot y_{B}.
  • Unlike Mixup (which blends pixel values everywhere and creates 'ghost' images), CutMix keeps local pixel statistics intact — better for localization tasks and dense prediction.
  • Often combined with Mixup in modern vision recipes.
Permalink

What is RandAugment and why is it a nice augmentation policy?

medium
  • Instead of learning an augmentation policy (AutoAugment), RandAugment randomly picks N transforms from a fixed pool (rotate, shear, color jitter, ...) each with a shared magnitude M.
  • Two hyperparameters (N, M) instead of dozens — much easier to tune, and matches AutoAugment's accuracy on ImageNet.
  • TrivialAugment (2021) simplifies further: apply one random transform with a random magnitude.
Permalink

What is Test-Time Augmentation (TTA)?

easy
  • At inference, apply K augmentations (crops, flips, color jitter) to the same input, run K forward passes, and average the predictions.
  • Better probability estimates, often 0.5-2% accuracy gain in classification and detection, at K× inference cost.
  • Popular in Kaggle competitions and medical imaging.
  • Not free — production usually skips it or uses only 2-4 augmentations.
Permalink
10

Convolutional networks

11 concepts

What inductive biases do CNNs have?

medium
  • (1) Locality: convolutions look at small neighborhoods.
  • (2) Translation equivariance: shifting the input shifts the feature map.
  • (3) Weight sharing: same filter applied everywhere, so few parameters.
  • (4) Hierarchy via stacking: early layers learn edges, later layers learn parts and objects.
  • These biases make CNNs data-efficient for images compared to plain MLPs.
Permalink

What is the receptive field in a CNN?

medium
  • The receptive field of a neuron is the region of the input image that influences its value.
  • It grows with depth, kernel size, and stride/dilation.
  • Deeper layers see larger regions.
  • Techniques to grow the receptive field: stacking layers, larger kernels, strided convolutions, dilated (atrous) convolutions, and pooling.
  • A large enough receptive field is required to capture global context.
Permalink

SAME vs VALID vs REFLECT padding — what are the practical differences?

easy
  • VALID = no padding (output shrinks each layer).
  • SAME = pad zeros so output has same H, W as input for stride 1.
  • REFLECT = pad by reflecting pixels near the boundary (avoids the 'black-edge' artifacts you get with zero-padding in segmentation / super-resolution).
  • REPLICATE = pad by repeating the edge pixel.
  • Choice matters for boundary quality: SAME/zero is fine for classification, REFLECT/REPLICATE for dense prediction.
Permalink

What is a dilated (atrous) convolution?

medium
  • Insert (d-1) zeros between kernel elements to enlarge the receptive field without adding parameters or reducing resolution.
  • A 3×3 kernel with dilation 2 has an effective 5×5 view.
  • Standard trick in segmentation networks (DeepLab, HRNet) where downsampling would lose spatial detail.
  • Stacked dilated convs with growing dilation cover a large receptive field cheaply — also used in WaveNet for 1D audio.
Permalink

What is a transposed convolution ('deconv') and its checkerboard artifact issue?

hard
  • A convolution that upsamples by inserting zeros between pixels then applying a normal conv — output is larger than input.
  • Used in decoders (U-Net upsampling), GAN generators, and dense prediction.
  • Checkerboard artifacts occur when kernel size isn't divisible by stride: some output pixels get contributions from more kernel positions than others.
  • Fix: use bilinear upsample + normal 3x3 conv, or ensure kernelsize  =  2  ⋅  stride\mathrm{kernel}_{\mathrm{size}}\; = \;2\; \cdot \;\mathrm{stride}.
Permalink

How does a depthwise separable convolution reduce compute?

medium
  • Factor a standard conv into (1) depthwise: one k×k filter per input channel independently; (2) pointwise: a 1×1 conv mixing the depthwise outputs across channels.
  • Compute goes from Cin⋅Cout⋅k⋅kC_{\mathrm{in}} \cdot C_{\mathrm{out}} \cdot k \cdot k to Cin⋅k⋅k  +  Cin⋅CoutC_{\mathrm{in}} \cdot k \cdot k\; + \;C_{\mathrm{in}} \cdot C_{\mathrm{out}} — for k=3, Cin=Cout=512C_{\mathrm{in}} = C_{\mathrm{out}} = 512, that's 8-9x cheaper.
  • Foundation of MobileNet, Xception, EfficientNet.
  • Small accuracy loss vs standard convs but huge speedups on mobile / embedded.
Permalink

What are 1x1 convolutions used for?

easy
  • Three roles: (1) channel-mixing / channel-wise projection — a matrix multiply across channels per spatial location, so it's a cheap way to change the number of channels; (2) bottleneck in ResNet blocks (reduce channels → 3×3 conv → expand channels) to save compute; (3) pointwise conv in depthwise-separable architectures.
  • Effectively a per-position linear layer.
Permalink

Max pooling vs average pooling — when do you pick each?

easy
  • Max pool: keeps the strongest activation in the window — sharper features, invariance to small translations, dominant in classification CNNs.
  • Average pool: smoother, retains overall energy — better in generative / decoder paths where you don't want to lose detail.
  • Modern architectures often replace pool with strided convs (learnable downsampling).
  • Global average pool (GAP) at the end of a CNN replaces the fully-connected head and reduces overfitting.
Permalink

Why does Global Average Pooling replace the FC head in modern CNNs?

medium
  • Take the mean of each feature map across H, W to get a C-dim vector, then apply a linear classifier.
  • Advantages: (1) drastic parameter reduction — no huge FC on flattened features; (2) intrinsic regularization; (3) enables class activation maps (CAM) since one channel per class is aligned with a class score; (4) works with any input resolution.
  • Introduced in Network-in-Network and popularized by ResNet.
Permalink

What is adaptive pooling and why is it useful?

easy
  • Given a target output size (Hout,  Wout)(H_{\mathrm{out}}, \;W_{\mathrm{out}}), computes the kernel and stride needed to produce that output regardless of input size.
  • Standard trick to accept variable input resolutions with a fixed-size classifier head. torch.nn.AdaptiveAvgPool2d((1,1)) is the go-to for global pooling.
  • Also useful for detection networks that need fixed-size feature maps for RoI heads.
Permalink

What is ConvNeXt's philosophy?

hard
  • Liu et al. (2022) 'modernize' a ResNet using tricks from Swin: bigger kernels (7×7 depthwise), GELU + LayerNorm, inverted bottlenecks, fewer activations / norms, and modern training recipes (AdamW + Mixup + RandAugment).
  • The result is a pure-conv architecture that matches Swin Transformer on ImageNet, detection, and segmentation.
  • Shows attention isn't magical — modern training + design does most of the work.
Permalink
11

CNN architectures

27 concepts

What made AlexNet (2012) a breakthrough?

easy
  • AlexNet: 8 layers, ReLU activations (huge speedup vs sigmoid), dropout (novel regularization), overlapping max pooling, local response normalization (LRN, now obsolete), heavy data augmentation, and GPU training split across 2 GTX 580s.
  • Won ImageNet 2012 by 10+ percentage points — kicked off the deep-learning revolution and made GPUs mandatory for vision.
Permalink

What is the design principle behind VGG?

easy
  • Stack 3x3 conv blocks repeatedly (2, 3, or 4 in a row) and downsample with max-pooling.
  • Two stacked 3x3 convs have the same receptive field as one 5x5 but with fewer parameters and one extra nonlinearity — makes deeper nets with more nonlinearity cheap.
  • VGG-16 / VGG-19 became the standard feature extractor for years.
  • Downsides: enormous FC layers (140M params in VGG-16).
Permalink

What is the Inception module?

medium
  • Multi-branch block that applies 1x1, 3x3, 5x5 convolutions and 3x3 max pool in parallel, concatenates the outputs along channels.
  • Each branch captures features at a different scale. 1x1 bottlenecks reduce cost.
  • Also introduces auxiliary classifiers at intermediate depths for training deep nets.
  • GoogLeNet (Inception v1) won ImageNet 2014 with 22 layers and 6M params — 12x smaller than VGG-16.
Permalink

What is the key insight of ResNet's identity mapping?

medium
  • A residual block computes y = F(x) + x, where F is a stack of convolutions.
  • If the optimal function is close to identity, the block only needs to learn a small correction F(x) ≈ 0 — easier than learning the identity from scratch.
  • Enabled training 152-layer nets on ImageNet in 2015.
  • Skip connections also let gradients flow directly to earlier layers → no vanishing.
Permalink

How is DenseNet different from ResNet?

medium
  • In a DenseNet block, each layer receives the concatenated feature maps of every preceding layer, not just the previous one.
  • Massive feature reuse — each layer produces only 'growth rate' k channels (typically 12-32), so the whole network has surprisingly few parameters.
  • Trade-offs: high memory usage (long concatenated feature maps), slower on GPU.
  • Cool idea but ResNet-derived architectures dominate today.
Permalink

What is MobileNet designed for?

medium
  • Efficient inference on mobile and embedded devices.
  • Uses depthwise-separable convolutions everywhere and a 'width multiplier' α scaling all layer widths, plus a resolution multiplier ρ.
  • MobileNet-v2 adds inverted residuals (expand → depthwise → project) and linear bottlenecks.
  • MobileNet-v3 uses NAS + h-swish. 5-10x smaller and faster than ResNet at similar accuracy.
Permalink

What is compound scaling in EfficientNet?

hard
  • Scale depth (d), width (w), and resolution (r) together by a single coefficient φ: d = α^φ, w = β^φ, r = γ^φ, with α⋅β2⋅γ2{\alpha} \cdot {\beta}^{2} \cdot {\gamma}^{2} ≈ 2 to keep FLOPs proportional to 2^φ.
  • Better than scaling one dimension at a time.
  • Combined with a NAS-found base (EfficientNet-B0) and MBConv blocks (mobile inverted bottleneck), EfficientNet-B7 hit 84% ImageNet top-1 in 2019 at 8.4x fewer params than the previous SOTA.
Permalink

How does a Vision Transformer (ViT) treat images?

hard
  • Split image into fixed-size patches (e.g., 16×16), linearly embed each patch (like word embeddings), add positional embeddings, prepend a learnable [CLS] token, and feed to a standard transformer encoder.
  • The CLS token's final representation is the image feature for classification.
  • Requires huge pretraining data (JFT-300M) to match CNN accuracy — with less data, CNNs still win.
  • DeiT (Touvron 2021) added distillation + augmentation to make ViT competitive at ImageNet-scale.
Permalink

What does Swin Transformer do differently from ViT?

hard
  • Swin uses hierarchical (multi-scale) feature maps like CNNs, with attention computed inside local windows (e.g., 7×7 patches).
  • Adjacent transformer blocks shift the windows so information flows across window boundaries — hence 'Swin' (Shifted Windows).
  • O(N) complexity in image size vs ViT's O(N2)O(N^{2}), and produces multi-scale features suitable for detection / segmentation.
Permalink

How does a two-stage detector (Faster R-CNN) work?

hard
  • Stage 1: a Region Proposal Network (RPN) sliding a small conv over the backbone's feature map generates ~1k-2k class-agnostic object proposals (anchor boxes + objectness).
  • Stage 2: RoIAlign extracts fixed-size features from each proposal, then a small head classifies the region and refines the box.
  • Slower but more accurate than one-stage detectors — still SOTA on many benchmarks.
  • Feature Pyramid Networks (FPN) extend it to multi-scale features.
Permalink

How does YOLO / SSD / RetinaNet differ from two-stage detection?

medium
  • One-stage detectors predict class and box in a single forward pass over dense anchor / grid locations — no proposal step.
  • YOLO: divide image into an S×S grid, each cell predicts B boxes + class probs.
  • SSD: multi-scale feature maps with different anchor scales.
  • RetinaNet: adds Focal Loss to fix the extreme class imbalance (background vs objects).
  • Much faster, real-time capable; historically less accurate but the gap narrowed with recent versions.
Permalink

What is Non-Maximum Suppression (NMS) and its variants?

medium
  • Post-processing that removes duplicate detections: sort boxes by confidence, keep the top one, remove all others with IoU > threshold with it, repeat.
  • Standard threshold: 0.5.
  • Variants: Soft-NMS (lower the score of overlapping boxes instead of removing, better for occluded objects), matrix NMS (used in SOLOv2 segmentation), and class-aware vs class-agnostic.
  • Recent detectors (DETR) avoid NMS entirely via set prediction.
Permalink

What are anchor boxes and their downsides?

medium
  • Predefined reference boxes of various sizes and aspect ratios placed at every location of the feature map.
  • The network predicts offsets and class scores relative to each anchor.
  • Downsides: (1) requires careful anchor design (scales, ratios) per dataset; (2) massive class imbalance (foreground:background can be 1:1000); (3) many hyperparameters.
  • Anchor-free methods (FCOS, CenterNet, DETR) predict box centers or points directly and skip anchors.
Permalink

Why is U-Net so popular for segmentation?

medium
  • Encoder-decoder with skip connections between symmetric layers.
  • Encoder downsamples via convs + pooling; decoder upsamples via transposed convs / bilinear + convs, and concatenates the same-resolution encoder features.
  • Skips preserve fine spatial detail through the low-resolution bottleneck.
  • Simple, data-efficient (works on small medical datasets), still the go-to baseline for biomedical / satellite / cell segmentation.
Permalink

How does Mask R-CNN extend Faster R-CNN for instance segmentation?

hard
  • Add a parallel FCN mask-prediction head to each RoI, predicting a small binary mask (28×28 typically) per class.
  • Replaces RoIPool with RoIAlign (bilinear interpolation instead of quantization) — critical for precise mask localization.
  • Preserves the classification / box head.
  • Standard instance-segmentation baseline; adopted broadly for pose estimation, keypoints, panoptic segmentation.
Permalink

What is a Feature Pyramid Network (FPN)?

hard
  • A top-down architecture that fuses features from multiple backbone stages (different resolutions and semantic levels) into a pyramid where each level is high-resolution and semantically rich.
  • Take the deep low-resolution feature, upsample, add to the mid-resolution feature, and so on.
  • Powers most modern detectors (Faster R-CNN + FPN, RetinaNet, YOLOv3+, Mask R-CNN) by giving each RoI head or anchor set the right-scale features.
Permalink

How does DETR reformulate object detection?

hard
  • DETR (Carion 2020) treats detection as a set-prediction problem: a transformer decoder outputs a fixed set of N (>>true count) predictions in parallel, matched to ground-truth objects via bipartite Hungarian matching during training.
  • No anchors, no NMS.
  • Simple, elegant, but slow to converge (500+ epochs) and struggles on small objects.
  • Deformable-DETR fixes both by using deformable attention and multi-scale features.
Permalink

Describe a single transformer encoder block in detail.

hard
  • Input x.
  • Sublayer 1: y = x + Dropout(MultiHeadAttention(LayerNorm(x))).
  • Sublayer 2: z = y + Dropout(MLP(LayerNorm(y))), where MLP = Linear → GELU → Linear with dffn  =  4d_{\mathrm{ffn}}\; = \;4d.
  • Modern variants use RMSNorm instead of LN, GLU-family MLPs, and pre-norm placement.
  • All ops are parallel across sequence positions — no recurrence.
Permalink

How does a decoder block differ from an encoder block?

hard
  • Decoder has three sublayers: (1) masked self-attention (each token can only attend to earlier positions — causal mask); (2) cross-attention over encoder outputs (queries from decoder, keys/values from encoder); (3) MLP.
  • Each still wrapped in pre-norm + residual + dropout.
  • Decoder-only LLMs (GPT) skip cross-attention and use only masked self-attention + MLP.
Permalink

Encoder-only vs decoder-only vs encoder-decoder — when do you pick each?

medium
  • Encoder-only (BERT, RoBERTa): bidirectional attention over the whole input → best for understanding tasks (classification, NER, retrieval embeddings).
  • Decoder-only (GPT, LLaMA): causal attention → best for open-ended generation and general chat assistants.
  • Encoder-decoder (T5, BART): full attention encoder + causal decoder + cross-attention → best for seq2seq (translation, summarization).
  • Modern LLMs converged on decoder-only for scale simplicity and general-purpose behavior.
Permalink

Explain focal loss and why it's used in dense detection.

hard
  • FL(pt)\mathrm{FL}(p_{t}) = -α  ⋅  (1  −  pt){\alpha}\; \cdot \;(1\; - \;p_{t})^γ  ⋅  log⁡(pt){\gamma}\; \cdot \;\operatorname{log}(p_{t}), where ptp_{t} is the model's probability for the true class.
  • The (1−pt)(1 - p_{t})^γ term (γ ~ 2) down-weights well-classified examples, keeping the loss focused on hard examples.
  • Fixes the extreme class imbalance in one-stage detectors (RetinaNet) where 100k background anchors dominate a few foreground ones.
  • Also useful for imbalanced classification.
Permalink

Write Dice loss and its role in segmentation.

medium
  • Dice = 1 - 2 * |P ∩ G| / (∣P∣  +  ∣G∣)( \mid P \mid \; + \; \mid G \mid ) — one minus the Dice coefficient (F1 on binary masks).
  • Directly optimizes overlap between predicted and ground-truth masks.
  • Robust to foreground/background imbalance in medical images where the target is 1% of pixels.
  • Often combined with BCE (Dice + BCE, or Dice + focal) for stability and calibration.
Permalink

What is perceptual loss and why is it better than pixel-wise for image quality?

medium
  • Loss computed as MSE (or cosine similarity) between features from a pretrained network (e.g., VGG-19 conv layers) applied to prediction and target, rather than on raw pixels.
  • Pretrained features capture perceptual similarity — small changes in features correspond to visually similar images.
  • Pixel-MSE encourages blurry averages of possible outputs; perceptual loss produces sharper, more realistic reconstructions.
  • Used in super-resolution, style transfer, image translation.
Permalink

What are the main ideas of StyleGAN?

hard
  • (1) Map input latent z to an intermediate 'style' space W with an MLP — disentangles factors of variation.
  • (2) Inject style at every layer of the generator via adaptive instance normalization (AdaIN) — controls features at multiple scales.
  • (3) Noise injection at each layer adds fine-grained stochastic detail (hair, freckles).
  • (4) Progressive growing (StyleGAN1) or top-down architecture (StyleGAN2/3) for high resolution.
  • Produces photorealistic 1024×1024 faces and enables style mixing.
Permalink

How does Latent Diffusion (Stable Diffusion) reduce compute?

hard
  • Train a VAE / autoencoder that maps 512×512 images to an 8-16x smaller latent space (e.g., 64×64×4).
  • Run the diffusion process in that latent space instead of pixels.
  • Compute drops ~50x, memory drops ~64x, quality is comparable to pixel-space diffusion after the VAE decoder up-samples the final latent.
  • Foundation of Stable Diffusion (Rombach 2022) — democratized text-to-image generation on consumer hardware.
Permalink

What are normalizing flows and their trade-off vs GANs / diffusion?

hard
  • Sequence of invertible transformations from a simple base (N(0, I)) to the data distribution, using the change-of-variables formula for exact log-likelihood.
  • Advantages: exact density evaluation, invertible sampling.
  • Disadvantages: constrained architectures (each layer must be invertible with a tractable Jacobian) → less expressive per-parameter than GAN / diffusion.
  • Used in density estimation and RL where exact log p is needed.
  • Modern generative modeling has mostly shifted to diffusion for quality.
Permalink

What is Neural Architecture Search (NAS)?

hard
  • Automated search for the best architecture (layers, connections, widths) given a target task and constraints (FLOPs, latency, memory).
  • Methods: reinforcement learning over architectures (NASNet, MnasNet), evolutionary search (AmoebaNet), gradient-based (DARTS — differentiable), or one-shot supernets (Once-For-All).
  • Produced EfficientNet, MobileNetV3, MnasNet — SOTA vision architectures under mobile budgets.
  • Now less used for LLMs (transformers are already close to optimal at scale); still active in edge vision and hardware-aware design.
Permalink
12

Object detection & segmentation

1 concept

What are IoU / Jaccard and Tversky losses?

hard
  • IoU (Jaccard) = |P ∩ G| / |P ∪ G|; loss = 1 - IoU.
  • Similar to Dice but stricter.
  • Tversky loss generalizes: T(P, G) = |P∩G| / (∣PG∣  +  α⋅∣PG∣  +  β⋅∣GP∣)( \mid PG \mid \; + \;{\alpha} \cdot \mid PG \mid \; + \;{\beta} \cdot \mid GP \mid ), where α, β control the penalty on false positives vs false negatives.
  • Handy for skewed segmentation where you want to trade recall vs precision. α=β=0.5 recovers Dice.
Permalink
13

Recurrent networks

8 concepts

Why did transformers replace RNNs for sequence modeling?

medium
  • RNNs process tokens sequentially, so they don't parallelize across time and struggle with long-range dependencies due to vanishing gradients (LSTMs help but only partly).
  • Transformers process all tokens in parallel with attention, capture arbitrary-range dependencies directly, and scale much better on GPUs/TPUs.
  • RNNs still appear in niche low-resource or online settings and in modern SSMs (Mamba) that revisit sequential modeling.
Permalink

Explain the four gates of an LSTM.

medium
  • (1) Forget gate f  =  σ(Wf  ⋅  [ht−1,  xt])f\; = \;{\sigma}(W_{f}\; \cdot \;[h_{t - 1}, \;x_{t}]): what to remove from cell state.
  • (2) Input gate i = σ(...) and candidate ctilde  =  tanh()c_{\mathrm{tilde}}\; = \;\mathrm{tanh}(): what to add.
  • (3) Cell update: ct  =  f  ⋅  ct−1  +  i  ⋅  ctildec_{t}\; = \;f\; \cdot \;c_{t - 1}\; + \;i\; \cdot \;c_{\mathrm{tilde}}.
  • (4) Output gate o = σ(...) and ht  =  o  ⋅  tanh(ct)h_{t}\; = \;o\; \cdot \;\mathrm{tanh}(c_{t}).
  • The additive cell-state update lets gradients flow across many timesteps without vanishing.
  • Introduced by Hochreiter & Schmidhuber 1997.
Permalink

GRU vs LSTM — how are they different in practice?

medium
  • GRU merges forget + input gates into an update gate and drops the separate cell state (uses only h).
  • Fewer parameters (~25% less), often trains faster, similar accuracy on most sequence tasks.
  • LSTM is a bit more expressive on very long sequences with complex temporal dynamics.
  • Rule of thumb: try GRU first for simplicity; use LSTM when GRU underfits.
Permalink

How does a bidirectional RNN work and when is it appropriate?

easy
  • Run two RNNs — one left-to-right, one right-to-left — and concatenate their hidden states at each timestep.
  • Provides access to both past and future context.
  • Appropriate for tagging tasks (POS, NER), speech recognition, classification.
  • Not usable for online generation (you need the full sequence).
  • Transformers with full attention subsume this by attending both directions in encoder-only setups (BERT).
Permalink

What was Bahdanau attention and why was it a big deal?

hard
  • Bahdanau et al. (2014) introduced attention for seq2seq: at each decoder step, compute alignment scores between the current decoder hidden state and every encoder hidden state, softmax to get weights, take a weighted sum to form a 'context' vector, and use it in the decoder update.
  • Removed the RNN's fixed-length bottleneck between encoder and decoder — enabled machine translation on long sentences.
  • Precursor to full self-attention.
Permalink

How does beam search work and what is its main failure mode?

medium
  • At each decoding step, keep the k highest-probability sequences ('beams') so far, expand each by one token, keep the top k of the resulting k*|V∣  candidatesV \mid \;\mathrm{candidates}.
  • Approximate search — better than greedy, cheaper than exhaustive.
  • Failure mode: length bias (shorter sequences get higher joint probability) — fix with length normalization (divide  by  lengthα)(\mathrm{divide}\;\mathrm{by}\;\mathrm{length}{\alpha}).
  • Also produces 'safe, boring' outputs on open-ended generation — LLM decoders now prefer sampling with temperature / top-p.
Permalink

Why is length normalization needed in beam search?

medium
  • Log-probability of a sequence is the sum of log P(tokent  ∣  history)P(\mathrm{token}_{t}\; \mid \;\mathrm{history}) — always negative, and longer sequences pile up more negatives.
  • Without normalization, beam search prefers shorter sequences by construction.
  • Length normalization divides by length^α (α ~ 0.6-1.0) or uses a length penalty.
  • Standard in NMT and summarization.
  • Modern LLM sampling uses temperature/top-p/top-k instead of beam search, so length norm is less critical there.
Permalink

What is CTC loss and where is it used?

hard
  • Connectionist Temporal Classification (Graves 2006): sums the probability over all valid alignments between an input sequence (frames) and an output sequence (labels) that may be shorter.
  • Allows a 'blank' token and repeated labels.
  • Used in speech recognition (DeepSpeech), handwriting recognition — anywhere you have variable-length outputs but no per-frame labels.
  • Modern speech models often replace CTC with attention-based or CTC+Transducer hybrids.
Permalink
14

Attention & transformers

22 concepts

How does self-attention work in a transformer?

hard
  • For each token, produce a Query, Key, and Value vector.
  • The output for token i is a weighted sum of all Values, where the weight is softmax⁡(Qi  ⋅  Kj  /  sqrt(dk))\operatorname{softmax}(Q_{i}\; \cdot \;K_{j}\; / \;\mathrm{sqrt}(d_{k})).
  • Multi-head attention runs this in parallel with different projections and concatenates the results.
  • Attention lets every token look at every other token — capturing long-range dependencies with O(n2)O(n^{2}) complexity.
Permalink

Why does a transformer need positional encoding?

hard
  • Self-attention is permutation-invariant — without position info, a sentence and its shuffle look identical.
  • Positional encoding injects order.
  • Options: fixed sinusoidal (original Transformer), learned absolute embeddings (BERT, GPT-2), relative positional bias (T5), and rotary position embeddings — RoPE — used in LLaMA, Mistral, and most modern LLMs.
Permalink

Why is attention scaled by 1/sqrt(dk)1 / \mathrm{sqrt}(d_{k})?

hard
  • Assume q, k have entries with mean 0 and variance 1 (post-init or post-normalization).
  • Their dot product has variance dkd_{k}.
  • As dkd_{k} grows (say, 64+), un-scaled dot products become large in magnitude, pushing softmax into the saturated regime where one weight is ~1 and the rest ~0 → tiny gradients w.r.t. the losing keys.
  • Dividing by sqrt(dk)\mathrm{sqrt}(d_{k}) rescales the variance back to 1 and keeps softmax gradients healthy.
Permalink

Why multi-head attention instead of a single big head?

medium
  • Different heads can attend to different types of relationships (syntactic, semantic, positional) in different subspaces.
  • Multi-head with H heads of dim d/H has similar compute to one head of dim d, but the diversity of learned attention patterns empirically helps a lot.
  • Also enables interpretation via per-head attention maps.
  • Typical: 8-32 heads for d ~ 512-4096.
Permalink

How do you pick the head dimension dheadd_{\mathrm{head}}?

hard
  • Common choice: dhead  =  d  /  nheadsd_{\mathrm{head}}\; = \;d\; / \;n_{\mathrm{heads}} (so total FLOPs match a single-head version with hidden dim d). dheadd_{\mathrm{head}} too small (< 32) → heads can't represent complex relationships; too large (> 128) → wastes parameters on redundancy.
  • Typical: dhead  =  64−128d_{\mathrm{head}}\; = \;64 - 128.
  • Grouped-query attention (GQA, MQA) share K/V across groups of heads to reduce KV cache without hurting quality much.
Permalink

Why is standard self-attention O(n2)O(n^{2}) in sequence length?

medium
  • For each of n query tokens, compute a dot product with each of n key tokens → n2n^{2} dot products, each of size d.
  • Then a softmax over n weights per query.
  • Both compute and memory are O(n2  ⋅  d)O(n^{2}\; \cdot \;d).
  • Fine at n = 512-2048; painful at n = 32k+.
  • Motivates efficient / sparse / linear-attention variants and paging tricks (Flash Attention doesn't reduce complexity but cuts memory dramatically via tiling).
Permalink

How do Performer / Linformer / linear attention reduce O(n2)O(n^{2})?

hard
  • Performer: replace exp⁡(q⋅k  /  sqrt(d))\operatorname{exp}(q \cdot k\; / \;\mathrm{sqrt}(d)) with a positive-feature kernel φ(q)·φ(k) that lets you re-associate the matmul as (KT  V)(K^{T}\;V)(QT)(Q^{T}) — O(n  ⋅  d2)O(n\; \cdot \;d^{2}) instead of O(n2  ⋅  d)O(n^{2}\; \cdot \;d).
  • Linformer: project the sequence-length dimension of K, V from n to a small constant k → O(n * k * d).
  • Trade-offs: approximation error, sometimes worse quality on long-range benchmarks vs full attention.
Permalink

What is sparse / sliding-window attention?

medium
  • Restrict each token to attend to a fixed neighborhood (window of size w) rather than all n tokens.
  • Complexity drops from O(n2)O(n^{2}) to O(n * w).
  • Stack layers so the effective receptive field grows with depth.
  • Used in Longformer (window + a few global tokens), BigBird, Mistral's sliding window attention.
  • Great when relevant context is local.
  • Combine with a few globally-attending tokens for long-range info.
Permalink

How do Longformer / BigBird combine sparse and global attention?

hard
  • Combine local sliding-window attention (each token sees a window of size w) with a handful of 'global' tokens (e.g., [CLS], question tokens in QA) that attend to every position and are attended by every position.
  • BigBird adds a random-attention pattern for provably close approximation of full attention.
  • Enables 4k-16k context on hardware where full attention would OOM.
Permalink

What is Flash Attention?

hard
  • Dao et al. (2022) IO-aware attention algorithm: tile Q, K, V into blocks that fit in on-chip SRAM, compute softmax incrementally with the online softmax trick, and never materialize the full n×n attention matrix in HBM.
  • Same math as standard attention (exact, not approximate), but 2-4x faster and O(n) memory instead of O(n2)O(n^{2}).
  • Enabled longer contexts (32k, 100k) on the same hardware.
  • Flash-Attention-2 improves parallelism further.
Permalink

What is the KV cache in transformer inference?

hard
  • During autoregressive generation, tokens are produced one at a time.
  • For each layer, the K and V projections of all previously generated tokens are cached; only the new token's Q, K, V are computed each step.
  • Time per new token drops from O(n2)O(n^{2}) to O(n).
  • Memory cost: 2  ⋅  nlayers  ⋅  nheads  ⋅  dhead  ⋅  seqlen2\; \cdot \;n_{\mathrm{layers}}\; \cdot \;n_{\mathrm{heads}}\; \cdot \;d_{\mathrm{head}}\; \cdot \;\mathrm{seq}_{\mathrm{len}} per sequence — often dominates GPU memory in long-context serving.
  • Techniques: MQA/GQA (share K/V across heads), Paged attention (vLLM), sliding-window caches.
Permalink

Explain sinusoidal positional encoding.

medium
  • PE(pos,  2i)  =  sin(pos  /  10000(2i/d))\mathrm{PE}(\mathrm{pos}, \;2i)\; = \;\mathrm{sin}(\mathrm{pos}\; / \;10000(2i / d)); PE(pos,  2i+1)  =  cos(pos  /  10000(2i/d))\mathrm{PE}(\mathrm{pos}, \;2i + 1)\; = \;\mathrm{cos}(\mathrm{pos}\; / \;10000(2i / d)).
  • Each dimension is a sinusoid of a different frequency.
  • Advantage: encodes relative position (offset by k is a linear map on the encoding) and extrapolates to longer sequences than seen at training.
  • Used in the original Transformer paper.
  • Simpler learned-embedding variants replaced it in BERT / GPT-2 but modern LLMs favor RoPE / ALiBi.
Permalink

What is the main limitation of learned absolute positional embeddings?

medium
  • They can't extrapolate — at inference, positions beyond max⁡positionembeddings\operatorname{max}_{\mathrm{position}}\mathrm{embeddings} have never been trained and produce garbage.
  • Also, they encode absolute rather than relative positions, so translating the sentence within a longer context changes the embedding.
  • Solved by relative PE (Shaw), RoPE (rotational), or ALiBi (linear bias on attention scores) — all of which behave well at longer sequences.
Permalink

How does relative positional encoding (Shaw / T5) work?

hard
  • Instead of adding a positional vector to the token embedding, inject position information into the attention score itself as a learned bias depending on the offset (i - j) between query and key.
  • In T5, this bias is bucketed (log-spaced) so the number of learned parameters stays finite.
  • Handles arbitrarily long sequences reasonably well, but requires custom attention kernels — less common than RoPE in modern LLMs.
Permalink

How does ALiBi encode position?

hard
  • Attention with Linear Biases (Press et al., 2021): add a linear penalty proportional to distance to the attention logits: ai,ja_{i, j} -= mh  ⋅  (i  −  j)m_{h}\; \cdot \;(i\; - \;j), where mhm_{h} is a fixed per-head slope.
  • No learned embeddings — just a static penalty.
  • Extrapolates naturally to longer sequences at inference than at training.
  • Simple, effective, used in BLOOM and some other LLMs; RoPE dominates elsewhere.
Permalink

How does Rotary Position Embedding (RoPE) work?

hard
  • Rotate the Q and K vectors in each 2D subspace of the head dimension by an angle proportional to the position: R(θpos)  ⋅  qR({\theta}_{\mathrm{pos}})\; \cdot \;q, R(θpos)  ⋅  kR({\theta}_{\mathrm{pos}})\; \cdot \;k, with θpos  =  pos  /  base{\theta}_{\mathrm{pos}}\; = \;\mathrm{pos}\; / \;\mathrm{base}^(2i/d).
  • Because of the rotation identity, (R(θi)q)(R({\theta}_{i})q) · (R(θj)k)(R({\theta}_{j})k) depends only on the difference θi  −  θj{\theta}_{i}\; - \;{\theta}_{j} — encodes relative position multiplicatively.
  • Extrapolates well (with NTK / linear scaling tweaks), used in LLaMA, Mistral, Falcon, Qwen, GPT-NeoX, most modern open LLMs.
Permalink

How does cross-attention differ from self-attention?

medium
  • Cross-attention: queries come from one sequence (e.g., decoder tokens), keys/values from another (e.g., encoder outputs, retrieved documents, image patches).
  • The decoder 'reads' the encoder representation without needing its own tokens to encode it.
  • Used in NMT decoder (attend to encoder), CLIP-style multimodal retrieval, retrieval-augmented generation, image captioning.
  • Self-attention has Q, K, V all from the same sequence.
Permalink

How much can you interpret a model from attention weights?

hard
  • Attention weights are a noisy signal for interpretability.
  • High weight ≠ 'the model uses this token'.
  • Multiple heads look at overlapping things; residual streams carry lots of info regardless of attention.
  • Better tools: attribution methods (integrated gradients, attention rollout), probing linear classifiers on hidden states, mechanistic interpretability (Anthropic's circuits).
  • Attention maps are a starting point, not a proof.
Permalink

Briefly, what do BLEU and ROUGE measure?

medium
  • BLEU: n-gram precision of the generated hypothesis against one or more reference translations, with a brevity penalty for short outputs.
  • Standard for machine translation.
  • ROUGE: n-gram recall of the summary against reference summaries (ROUGE-N for n-grams, ROUGE-L for longest common subsequence).
  • Standard for summarization.
  • Both are crude — modern metrics use learned embeddings (BERTScore, BLEURT, COMET) or LLM-as-judge.
Permalink

What are NTK-aware RoPE scaling and YaRN?

hard
  • Techniques to extend a RoPE-based LLM's context window beyond the training length without retraining from scratch.
  • Naive linear interpolation of positions (Position Interpolation) shrinks the effective frequency and hurts short-range attention.
  • NTK-aware scaling adjusts higher frequencies less than lower ones (motivated by the Neural Tangent Kernel view).
  • YaRN (Peng et al., 2023) refines this with per-frequency temperature and a small fine-tune — lets LLaMA-2 extend from 4k to 128k tokens at minimal quality loss.
Permalink

How do you extend a transformer to very long contexts?

hard
  • Options: (1) sparse/local attention (Longformer, Mistral's sliding window); (2) linear-attention approximations (Performer, RWKV, Mamba SSM); (3) retrieval augmentation (chunk documents, retrieve top-k with a vector store, stuff into the prompt); (4) position-encoding extrapolation (RoPE scaling, NTK-aware interpolation, YaRN); (5) hierarchical models (encode chunks separately then attend at a higher level).
  • Combine multiple techniques in practice.
Permalink

Attention is quadratic in sequence length. Why is FlashAttention still a major win without changing that?

hard
  • Because the practical bottleneck is memory traffic, not arithmetic.
  • A naive implementation materializes the full attention matrix in high-bandwidth memory, so the cost is dominated by writing and reading a quadratic-sized intermediate.
  • FlashAttention tiles the computation, keeps blocks in on-chip memory, and computes the softmax in a streaming, numerically stable way, so the quadratic matrix never exists in memory at all.
  • Complexity stays quadratic in time but memory becomes linear in sequence length, and wall-clock speed improves several-fold because the kernel is no longer memory-bound.
  • It is exact, unlike sparse or low-rank approximations, which is why it was adopted universally rather than as a quality tradeoff.
Permalink
15

Losses for deep learning

11 concepts

Write the Huber loss and explain when to use it.

medium
  • L_δ(y,  )  =  0.5⋅(y−)2(y, \;)\; = \;0.5 \cdot (y - )^{2} if  ∣y\mathrm{if}\; \mid y-ŷ| ≤ δ, else δ⋅(∣y−∣  −  0.5⋅δ){\delta} \cdot ( \mid y - \mid \; - \;0.5 \cdot {\delta}).
  • Quadratic for small errors (smooth, MSE-like gradients), linear for large errors (robust to outliers). δ ~ 1-2*σ of the noise.
  • Common in bounding-box regression (Smooth L1 with δ=1 in R-CNN family) and robust regression.
Permalink

What is log-cosh loss and its advantage over Huber?

medium
  • L(y, ŷ) = log(cosh(y - ŷ)) ≈ 0.5⋅(y−)20.5 \cdot (y - )^{2} for small errors and  ∣y\mathrm{and}\; \mid y-ŷ| - log⁡(2)\operatorname{log}(2) for large — same behavior as Huber but smooth everywhere (all derivatives exist).
  • No δ hyperparameter.
  • Slightly more expensive to compute.
  • Use it when you want Huber's outlier-robustness without tuning δ, and gradient smoothness matters (e.g., second-order optimizers).
Permalink

Write binary cross-entropy for probability p̂ and label y ∈ {0, 1}.

easy
  • L(y, p̂) = -[y*log(p̂) + (1-y)*log(1-p̂)].
  • Comes from the negative log-likelihood of a Bernoulli.
  • Always non-negative, zero iff p̂=y ∈ {0,1}.
  • In practice, apply the numerically-stable BCEWithLogitsLoss (input is logits, sigmoid + BCE fused with log-sum-exp for stability) — never compute sigmoid then log separately in fp16.
Permalink

Multi-class cross-entropy vs one-vs-rest — pros and cons?

medium
  • Cross-entropy on softmax: mutually exclusive classes, single loss, joint probability calibration → standard for classification.
  • One-vs-rest with K sigmoids + BCE: allows multi-label, imbalanced-per-label handling, per-class thresholding at inference.
  • Cross-entropy is more parameter-efficient for many classes; OVR is essential for multi-label but gives up joint calibration.
Permalink

Write the basic contrastive loss and when it's used.

medium
  • L  =  y  ⋅  d2  +  (1−y)  ⋅  max⁡(margin  −  d,  0)2L\; = \;y\; \cdot \;d^{2}\; + \;(1 - y)\; \cdot \;\operatorname{max}(\mathrm{margin}\; - \;d, \;0)^{2}, where y=1 for a positive pair (same class) and y=0 for a negative pair, d is a distance in the embedding space.
  • Pulls positives together, pushes negatives apart at least by a margin.
  • Used in Siamese nets, verification (faces, signatures), and metric learning.
  • Modern SSL uses info-NCE (below) instead.
Permalink

What is triplet loss and its main challenge?

hard
  • L(a, p, n) = max(d(a, p) - d(a, n) + margin, 0), where a=anchor, p=positive, n=negative.
  • Wants the anchor closer to positives than to negatives by at least the margin.
  • Main challenge: 'triplet mining' — most random triplets are easy (already satisfied), so you must actively sample semi-hard / hard triplets to keep training progressing.
  • Standard in face recognition (FaceNet).
  • Superseded in most tasks by info-NCE / softmax-based losses.
Permalink

Write info-NCE / NT-Xent and its role in SSL.

hard
  • L(x,  x+,  x−)L(x, \;x^{+}, \;x^{-}) = -log⁡[exp⁡(sim(x,  x+)  /  τ)  /  (exp⁡(sim(x,  x+)  /  τ)  +  Σ  exp⁡(sim(x,  x−)  /  τ))]\operatorname{log}[\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; / \;(\operatorname{exp}(\mathrm{sim}(x, \;x^{+})\; / \;{\tau})\; + \;{\Sigma}\;\operatorname{exp}(\mathrm{sim}(x, \;x^{-})\; / \;{\tau}))], sim = cosine similarity, τ = temperature.
  • Treats contrastive learning as a classification problem: 'which of these candidates is the positive?'.
  • Foundation of SimCLR, MoCo, CLIP, GPT-style retrieval.
  • Larger negative batches → richer learning signal.
Permalink

What is the standard reconstruction loss for autoencoders?

easy
  • MSE (or L1) between input and reconstruction for continuous data.
  • Binary cross-entropy per pixel for images normalized to [0, 1] (treats each pixel as a Bernoulli).
  • VAEs add a KL divergence between the posterior over latents and a prior — the ELBO objective.
  • Modern generative models use perceptual losses (feature-space MSE) or adversarial losses for sharper reconstructions.
Permalink

Write the vanilla GAN's minimax objective.

hard
  • min⁡G\operatorname{min}_{G} max⁡D\operatorname{max}_{D} Ex[log⁡  D(x)]  +  Ez[log⁡(1  −  D(G(z)))]E_{x}[\operatorname{log}\;D(x)]\; + \;E_{z}[\operatorname{log}(1\; - \;D(G(z)))].
  • D tries to output 1 for real, 0 for fake; G tries to make D output 1 on its fakes.
  • In practice, G is trained with the 'non-saturating' loss  −Ez[log⁡  D(G(z))]\mathrm{loss}\; - E_{z}[\operatorname{log}\;D(G(z))] (stronger gradients when D is confident).
  • WGAN replaces this with a Wasserstein-distance-based loss for better stability.
Permalink

What does WGAN + gradient penalty change vs vanilla GAN?

hard
  • WGAN uses the Wasserstein distance approximation: max⁡D\operatorname{max}_{D} E[D(x)] - E[D(G(z))] where D (the 'critic') must be 1-Lipschitz.
  • Original WGAN clipped weights (crude).
  • WGAN-GP (Gulrajani 2017) enforces Lipschitz softly with a gradient penalty λ  ⋅  (∣∣∇x  D(x)∣∣2  −  1)2{\lambda}\; \cdot \;( \mid \mid \nabla x\;D(x) \mid \mid _{2}\; - \;1)^{2} on interpolated points x̂.
  • Result: stable training, meaningful loss curve (correlated with sample quality), fewer mode-collapse issues.
Permalink

Write the VAE ELBO and explain each term.

hard
  • ELBO(x)  =  Eq(z∣x)\mathrm{ELBO}(x)\; = \;E_{q}(z \mid x)[log⁡  p(x∣z)]  −  KL⁡(q(z∣x)  ∣∣  p(z))[\operatorname{log}\;p(x \mid z)]\; - \;\operatorname{KL}(q(z \mid x)\; \mid \mid \;p(z)).
  • Reconstruction term: how well the decoder reproduces x from a latent sampled by the encoder.
  • KL term: pushes the encoder's posterior q(z∣x)q(z \mid x) toward the prior p(z) (usually N(0, I)).
  • Maximizing ELBO ≤ log p(x).
  • The KL acts as regularizer — without it, VAE collapses to an autoencoder. β-VAE weights the KL term for disentanglement.
Permalink
16

Transfer learning & fine-tuning

3 concepts

What is transfer learning and when is fine-tuning better than feature extraction?

medium
  • Transfer learning reuses a model pretrained on a large source task on a new target task.
  • Feature extraction freezes the backbone and trains a new head — fast, works when the target is small and similar to pretraining data.
  • Full fine-tuning updates all weights — better when you have enough target data or when target differs significantly.
  • Modern LLMs often use parameter-efficient fine-tuning (LoRA, adapters) to combine both benefits.
Permalink

What is LoRA and why is it the standard for parameter-efficient fine-tuning?

hard
  • LoRA (Hu et al., 2021) freezes the pretrained weights W and adds a low-rank update ΔW = B * A (with  A  ∈  R(r×d),  B  ∈  R(d×r),  r  <<  d)(\mathrm{with}\;A\; \in \;R(r \times d), \;B\; \in \;R(d \times r), \;r\; < < \;d).
  • Only A and B are trained → ~0.1-1% of the total parameters.
  • Matches full fine-tuning quality on most tasks at a fraction of memory and disk cost.
  • Multiple LoRA adapters can be swapped per task at inference.
  • QLoRA quantizes the frozen base to 4-bit and trains LoRA on top — fine-tune 65B models on a single 48GB GPU.
Permalink

With a small labelled dataset and a large pretrained backbone, which parameters do you actually train?

medium
  • Start with the smallest set that can express the task.
  • Train only a new head first, which is fast, cannot damage the backbone, and gives a baseline within minutes.
  • If that underfits, unfreeze the last block or two, since later layers hold the most task-specific features while early layers encode generic edges and textures that transfer.
  • Use a lower learning rate on unfrozen pretrained weights than on the fresh head, often by a factor of ten, to avoid destroying what pretraining learned.
  • With very little data, a parameter-efficient method such as low-rank adapters gives most of the benefit of full fine-tuning while touching a small fraction of the weights, which also makes the result cheap to store and easy to revert.
Permalink
17

Efficiency & distributed training

2 concepts

What is gradient checkpointing and its trade-off?

medium
  • Store activations at only a few 'checkpoint' layers instead of every layer.
  • During backward, recompute the missing activations from the nearest checkpoint on demand.
  • Memory drops from O(L) to O(sqrt(L)) with optimal checkpoints.
  • Trade-off: ~30% more compute in the backward pass.
  • Standard technique to fit larger models on limited GPUs (used in training large transformers).
Permalink

How does activation checkpointing interact with gradient accumulation and FSDP?

hard
  • Activation checkpointing (recompute activations in backward) trades ~30% extra compute for O(√L) memory.
  • Gradient accumulation reduces optimizer-step frequency but doesn't reduce peak activation memory.
  • FSDP shards parameters/gradients but activation memory stays the same.
  • All three are complementary — big LLM training uses checkpointing + gradient accumulation + FSDP + mixed precision + flash attention simultaneously to fit models 10x larger than would otherwise be possible.
Permalink
18

Generative models

4 concepts

What is mode collapse in GANs and how do you mitigate it?

hard
  • Generator maps most latents to a few output modes because those fool the discriminator well — trained distribution becomes much less diverse than the data.
  • Symptoms: repeated outputs, poor coverage of the true distribution.
  • Fixes: minibatch discrimination (D sees a batch and can penalize identical outputs), unrolled GAN (train D a few extra steps), WGAN-GP loss for smoother gradients, spectral norm on D, feature matching, or diffusion models (which don't suffer from mode collapse).
Permalink

Describe the forward and reverse processes in diffusion models.

hard
  • Forward (q): gradually add Gaussian noise to the data over T steps until it becomes ~ N(0, I).
  • Fixed, no learning.
  • Closed-form: xt  =  sqrt(αbart)⋅x0  +  sqrt(1  −  αbart)⋅ϵx_{t}\; = \;\mathrm{sqrt}(\alpha_{\mathrm{bar}}t) \cdot x_{0}\; + \;\mathrm{sqrt}(1\; - \;\alpha_{\mathrm{bar}}t) \cdot \epsilon.
  • Reverse (pθ)(p{\theta}): learn to denoise step-by-step, reversing q.
  • Train by minimizing EtE_{t},x0x_{0},eps [∣∣eps  −  epsθ(xt,  t)∣∣2][ \mid \mid \mathrm{eps}\; - \;\mathrm{eps}{\theta}(x_{t}, \;t) \mid \mid ^{2}] — the network predicts the noise added at each step.
  • At sampling, start from Gaussian noise and denoise for T (or fewer, with DDIM) steps.
Permalink

How does DDIM speed up diffusion sampling?

hard
  • DDIM (Song 2020) formulates a deterministic non-Markovian reverse process that shares the same training objective as DDPM but allows sampling with far fewer steps (e.g., 50 instead of 1000) with minimal quality loss.
  • Also enables deterministic sampling (identical noise → identical output), which is essential for tasks like image-to-image editing, inversion, and reproducibility.
  • Foundation for modern samplers (Euler, Heun, DPM-Solver++).
Permalink

What is classifier-free guidance (CFG) in diffusion?

hard
  • Train the model to accept both conditional (c) and unconditional (∅, e.g., empty prompt) inputs, by dropping the condition 10-20% of the time.
  • At sampling, form the guided prediction epsguided  =  eps\mathrm{eps}_{\mathrm{guided}}\; = \;\mathrm{eps}_θ(xt,  t,  )  +  w  ⋅  (epsθ(xt,  t,  c)  −  epsθ(xt,  t,  ))(x_{t}, \;t, \;)\; + \;w\; \cdot \;(\mathrm{eps}{\theta}(x_{t}, \;t, \;c)\; - \;\mathrm{eps}{\theta}(x_{t}, \;t, \;)). w>1 amplifies condition alignment (sharper prompt adherence) at some diversity cost.
  • Standard trick in Stable Diffusion / Imagen — biggest single quality lever after model size.
Permalink
19

Compression & interpretability

4 concepts

How does knowledge distillation work?

medium
  • Train a small 'student' network to match the outputs (usually soft probabilities at high temperature T) of a large pretrained 'teacher'.
  • Loss  =  α  ⋅  CE(student,  hardlabels)  +  (1−α)  ⋅  T2  ⋅  KL⁡(softmax⁡(student/T)  ∣∣  softmax⁡(teacher/T))\mathrm{Loss}\; = \;{\alpha}\; \cdot \;\mathrm{CE}(\mathrm{student}, \;\mathrm{hard}_{\mathrm{labels}})\; + \;(1 - {\alpha})\; \cdot \;T^{2}\; \cdot \;\operatorname{KL}(\operatorname{softmax}(\mathrm{student} / T)\; \mid \mid \;\operatorname{softmax}(\mathrm{teacher} / T)).
  • The soft targets carry more information than hard labels (they encode inter-class similarities), so the student often achieves 90-95% of the teacher's accuracy at 5-10x smaller size.
  • Foundational for model compression, mobile deployment, MLM speedups (DistilBERT).
Permalink

What are the main pruning techniques for neural networks?

medium
  • Unstructured pruning: set individual weights to zero based on magnitude (∣w∣  <  threshold)( \mid w \mid \; < \;\mathrm{threshold}) — high compression, but sparse matrices are hard to accelerate without specialized kernels.
  • Structured pruning: remove entire filters / channels / attention heads — smaller matrices, natively fast on GPUs.
  • Typical recipe: iterative magnitude pruning + fine-tuning to recover accuracy.
  • Modern LLM pruning: SparseGPT, Wanda — one-shot pruning post-training.
  • Combined with quantization for further compression.
Permalink

Post-training quantization vs Quantization-Aware Training — trade-offs?

hard
  • Post-training (PTQ): calibrate on a small dataset then quantize weights (and optionally activations) from fp32 to int8 / int4.
  • Fast (a few minutes), zero training overhead, some accuracy drop especially at ≤ 4 bits.
  • Quantization-aware training (QAT): simulate quantization noise during training (fake quant ops) so weights adapt — much smaller accuracy drop, especially at low bit widths, at the cost of a full fine-tuning run.
  • Popular libs: bitsandbytes, GPTQ, AWQ, LLM.int8().
Permalink

You need to halve inference cost. Do you quantize, prune, or distill?

medium
  • Quantize first, because it is the cheapest to try and the best understood.
  • Post-training quantization to 8-bit integers typically costs a fraction of a point of accuracy and needs only a small calibration set, and 4-bit weight-only quantization is now routine for language models.
  • Structured pruning gives real speedups only when the sparsity pattern matches what the hardware can exploit, since unstructured sparsity often saves memory without saving time.
  • Distillation gives the largest gains but is the most expensive, because it means training a new smaller model against the teacher's outputs, and it needs enough unlabelled data.
  • The pragmatic order is quantize, measure, then distil if you still need more, and prune only where the hardware rewards the pattern you can produce.
Permalink
You finished the lesson
Now test what stuck.
Interview flashcardsTake the quizBrowse other topics