43 interview questions on training dynamics, each answered in full. Free to read, no account needed.
What does Batch Normalization do?
medium- For each mini-batch, BN normalizes activations to zero mean and unit variance per feature, then applies learned scale and shift.
- Effects: faster convergence, higher learning rates, mild regularization, and reduced internal covariate shift.
- Downsides: depends on batch size, awkward for small batches, and behaves differently in train vs eval mode.
- LayerNorm (per-example) is preferred in transformers and RNNs.
How does dropout work and when is it applied?
easy- During training, dropout randomly zeros a fraction p of activations per forward pass, forcing the network to distribute knowledge and not co-adapt.
- At inference, all units are active and outputs are scaled (or scaling is done at train time — 'inverted dropout').
- Common rates: 0.1-0.5.
- Modern transformers use dropout in attention and MLP layers.
Why use a learning-rate schedule (warmup + cosine decay)?
medium- Large models with LayerNorm behave badly at high LR early on — warmup ramps LR from 0 to peak over a few thousand steps to avoid divergence.
- Then cosine (or linear) decay reduces LR smoothly toward zero, which improves final generalization.
- Standard recipe for training transformers and LLMs.
Why do residual (skip) connections help training deep networks?
easy- They provide a shortcut so gradients can flow directly back to earlier layers, alleviating vanishing gradients.
- They also make it easy for a layer to learn the identity function, so adding depth can't hurt.
- Residual blocks let us train 100+ layer networks (ResNet) and are used in transformers too (skip around attention and MLP).
What is gradient accumulation and when do you use it?
easy- Run several forward-backward passes with small mini-batches without calling optimizer.step(); sum the gradients.
- Then apply one optimizer step as if you had used a large batch.
- Simulates a large effective batch on limited GPU memory without changing statistics much (BN stats still per-microbatch — so combine with GroupNorm or SyncBN if BN is used).
How does batch size affect training dynamics?
medium- Larger batch → lower gradient noise, closer to true gradient, allows higher learning rate (roughly LR ∝ sqrt(batch)).
- But too large batches tend to converge to sharper minima that generalize slightly worse (the 'generalization gap').
- Small batches are noisier — regularizing effect that often helps final accuracy.
- Rule: scale LR with batch and use warmup to keep training stable at very large batches (LARS/LAMB for extreme scales).
What is deep double descent?
hard- Test error as a function of model capacity is not monotonically U-shaped: it goes up as capacity crosses the interpolation threshold (train error → 0), then goes down again for extremely overparameterized models.
- Documented in Belkin et al. (2019) and Nakkiran et al. (2020).
- Implication: in the overparameterized regime, bigger models can generalize better despite fitting the training data perfectly.
What is grokking in deep learning?
hard- Phenomenon (Power et al., 2022) where a network trained on a small algorithmic task memorizes the training set quickly (train loss → 0) but validation stays random for a long time — then, after many more epochs of continued training, suddenly generalizes to near-perfect validation accuracy.
- Suggests the network first memorizes, then slowly discovers a more compressed, generalizable circuit under continued regularization pressure.
In one sentence, what is the Lottery Ticket Hypothesis?
hard- Frankle & Carbin (2019): a randomly initialized dense network contains sparse subnetworks ('winning tickets') that — when trained in isolation from their original initialization — reach comparable accuracy in comparable or fewer iterations.
- Motivates pruning research: identify and train these tickets to shrink models dramatically without loss.
What is the 'implicit bias' of SGD?
hard- Among the infinite minimizers of the training loss in an over-parameterized net, SGD tends to select ones with certain 'flat' properties — approximately minimum-norm solutions in linear cases, and empirically flat-minimum / max-margin solutions in nonlinear ones.
- This implicit bias explains why deep nets trained by SGD generalize despite having capacity to fit noise.
- Weight decay and small LR reinforce this bias.
Your training loss becomes NaN. Debug it.
medium- (1) Check for divisions by zero and log of non-positive values — most common cause.
- (2) Reduce learning rate; loss NaN often means gradient exploded.
- (3) Add gradient clipping.
- (4) Verify data has no NaN / inf.
- (5) In fp16 / mixed precision, check the loss scale; use bf16 if available (more forgiving range).
- (6) Reset from a known-good checkpoint.
- (7) Print the loss and gradient norms every N steps in a rerun to isolate the offending step.
What problem does LAMB solve?
hard- LAMB (Layer-wise Adaptive Moments for Batch training) enables very large-batch training (32k+ per step) without loss of accuracy.
- It scales the Adam update per layer by the ratio ||θlayer|| / ||updatelayer||.
- Used to train BERT-Large in 76 minutes on TPU pods.
- Similar spirit to LARS (which does the same for SGD).
Describe the LR-range finder (Smith, 2015).
easy- Train the model for a few epochs with the learning rate increased exponentially from very small to very large.
- Plot loss vs LR.
- Pick a LR one order of magnitude below where the loss starts to diverge (or where the negative slope is steepest).
- Fast heuristic — replaces manual grid search over the most important hyperparameter.
- Available in fastai and PyTorch Lightning.
What is the one-cycle policy?
medium- Smith's 'super-convergence' recipe: linearly ramp LR up from LRmax/25 to LRmax over the first ~30% of training, then linearly decay back down.
- Simultaneously modulate momentum in the opposite direction (high when LR is low, low when LR is high).
- Enables very fast training with large max LR — regularizes via the 'LR-as-annealing-temperature' effect.
How should learning rate scale with batch size?
medium- Linear rule (Goyal et al., 2017): LR=baseLR⋅(batch/basebatch).
- Works up to ~8k batch in vision with sufficient warmup.
- Sqrt rule: LR ∝ sqrt(batch) — more conservative, sometimes better for Adam / transformers.
- Very large batches (>32k) need layer-wise scaling (LARS, LAMB).
- Always combine with warmup: large batches need longer warmup to avoid divergence.
What is gradient noise scale (GNS) and how do you use it?
hard- GNS (McCandlish et al., 2018) measures the ratio between the variance of the mini-batch gradient and its squared mean, roughly telling you how much batch size you can grow before returns diminish.
- Compute Bcrit=trace(Cov(g)) / ||g||².
- Below Bcrit, doubling batch halves steps.
- Above, benefits plateau.
- Guides choice of batch size for large-model training.
What is LARS and when is it used?
hard- Layer-wise Adaptive Rate Scaling (You et al., 2017).
- Scales the update per layer by ||θlayer|| / ||glayer||, keeping the ratio update-to-weight at ~1%.
- Enables training ResNet-50 in a few minutes with batch sizes up to 32k.
- Same idea as LAMB but for SGD-with-momentum.
- Standard for very-large-batch training in vision.
Why does a model with BatchNorm behave differently at train time vs eval time?
medium- Train: BN uses batch statistics (mean/var of the current mini-batch), providing regularization noise.
- Eval: uses stored running averages accumulated during training via EMA.
- If those running stats are out of sync with training (small batches, non-i.i.d. batches, incorrect running momentum), eval accuracy can drop dramatically.
- Common bug: fine-tuning with BN in train mode but small batches contaminates running stats.
What is stochastic depth / DropPath and where is it used?
medium- During training, randomly drop entire residual branches (set them to 0 and pass identity through the skip connection) with some probability p that increases with depth.
- Effectively trains an ensemble of networks of varying depth.
- Standard trick in modern deep vision transformers (Swin, ConvNeXt, DeiT) — enables much deeper training at higher accuracy.
What is early stopping and how do you configure it?
easy- Monitor a validation metric each epoch; stop training when it hasn't improved for 'patience' epochs.
- Restore the best-performing checkpoint.
- Prevents overfitting when training beyond the optimal point.
- Config: choose the right metric (accuracy for classification, F1 for imbalanced, loss for regression), patience ~5-20% of total epochs, and a small delta threshold to avoid stopping on noise.
What is Stochastic Weight Averaging (SWA)?
medium- During the last part of training (with a constant or cyclical LR), keep a running average of the weights: θswa=mean of θt across recent steps.
- Use θswa at inference.
- Averaging in the loss landscape moves toward the center of a flat minimum, which generalizes better than any individual point.
- Adds ~0 cost and often gains 0.5-1% accuracy.
- Extended to EMA of weights (used in EMA-teacher / EMA distillation).
How does snapshot ensembling work?
medium- Train with a cyclical LR schedule (SGDR / cosine with restarts).
- Save a checkpoint at each cycle's minimum LR (one snapshot per cycle).
- At inference, ensemble the K snapshots by averaging predictions.
- Same training cost as a single training run, but ensemble diversity comes from the LR restarts landing in different basins.
What is adversarial training (PGD, FGSM)?
hard- Generate small adversarial perturbations that fool the current model (FGSM = one gradient step; PGD = multiple projected gradient steps), then include them in training with the true labels.
- Improves robustness to bounded input perturbations at ~2-10x compute overhead.
- Trade-off: clean accuracy typically drops 3-5% while adversarial accuracy jumps from ~0% to 40-60%.
- Standard baseline for robust ML.
How are input images typically normalized for pretrained CNNs?
easy- Subtract ImageNet channel means [0.485, 0.456, 0.406] and divide by stds [0.229, 0.224, 0.225] (RGB).
- This matches the distribution the model saw during pretraining — using different stats can hurt accuracy noticeably during transfer.
- When training from scratch, either compute stats on your dataset or normalize to [-1, 1] / [0, 1].
- Store the norm inside the model's preprocess step so it can't be forgotten in production.
What is a solid default augmentation recipe for training a modern vision model?
medium- Random resized crop, horizontal flip, color jitter (brightness/contrast/saturation/hue), and one of RandAugment / TrivialAugment for stronger transforms.
- Add Random Erasing / Cutout.
- For classification, layer in Mixup (α=0.2) and CutMix (α=1.0).
- Label smoothing 0.1.
- Combined with AdamW + cosine schedule + EMA, this recipe gives you ~+2-4% over vanilla training on ImageNet and is the modern baseline (DeiT / ConvNeXt).
How should training resolution be chosen for a CNN or ViT?
medium- For CNNs: bigger resolution → higher accuracy up to a point (diminishing returns and OOM).
- Common choices: 224 for ResNet baselines, 256/288/380/456 for EfficientNet variants (compound scaling), 384/512 for ViT-Large.
- Train at moderate resolution, then fine-tune at higher resolution ('progressive resizing') for a cheap accuracy boost.
- For detection / segmentation, resolution matters more — small objects need high input res.
When do you use truncated BPTT and what's the trade-off?
medium- For very long sequences (language modeling on 1000+ tokens, time-series, audio).
- Unroll for k steps, backward, then carry hidden state forward but detach it from the graph.
- Bounded memory, faster training.
- Trade-off: can't learn dependencies longer than k without further tricks (e.g., overlap windows, memorized state).
- Modern transformers avoid this via full parallel attention over the whole context.
What is teacher forcing and its main pitfall?
medium- During training, feed the ground-truth previous token as input to the decoder at each step (instead of the model's own previous prediction).
- Speeds training and avoids compounding errors from incorrect early predictions.
- Pitfall: exposure bias — inference behavior differs from training (model has never seen its own errors).
- Fixes: scheduled sampling (mix true vs predicted with a schedule) or reinforcement fine-tuning.
What is scheduled sampling?
medium- Interpolate between teacher forcing and self-generation during training: with probability p feed the true token, with 1-p feed the model's own prediction.
- Decay p over training (start at 1, end near 0).
- Bridges the train/inference gap without full exposure.
- Works OK for RNN seq2seq but doesn't compose easily with parallel transformer training (which processes all positions at once).
Contrast the pretraining objectives of BERT, GPT, and T5.
medium- BERT: masked language modeling — mask 15% of tokens, predict them (bidirectional context).
- GPT: causal / next-token prediction — predict token t from tokens 1..t-1 (unidirectional).
- T5: 'span corruption' — replace random spans with sentinel tokens, decoder generates the missing spans (encoder-decoder).
- Each objective aligns with the target usage: BERT for embeddings, GPT for generation, T5 for text-to-text.
What is deep supervision / auxiliary loss?
medium- Attach small classification / regression heads to intermediate layers and add their losses to the main loss (with lower weights).
- Provides gradient signal deeper into the network — helps optimization of very deep nets and enables intermediate feature usefulness.
- Used in Inception (auxiliary classifiers), U-Net (deep supervision on multi-scale masks), and dense prediction.
- Modern residual nets need it less.
How does mixed-precision training work with fp16 / bf16?
medium- Store weights in fp32, cast to fp16 (or bf16) for forward and backward passes, accumulate gradients in fp32, then update weights in fp32.
- Cuts activation memory ~2x and speeds up compute 2-3x on modern GPUs with tensor cores. fp16 has small range (needs loss scaling to prevent gradient underflow); bf16 has fp32's range but less precision — usually plug-and-play, preferred on Ampere+.
- Use torch.cuda.amp / bf16 autocast.
Why does fp16 training need loss scaling?
hard- fp16 has a small dynamic range (~5e-8 to 6.5e4).
- Gradients often live near the underflow boundary — many gradient elements silently become zero.
- Solution: multiply loss by a scale factor S (e.g., 128) before backward → gradients are S× larger → survive fp16 range → divide by S before optimizer step.
- Dynamic loss scaling adjusts S automatically: increase when no NaN, halve when a NaN appears.
- Not needed with bf16 (has fp32's exponent range).
How does PyTorch DDP work under the hood?
medium- Each GPU has a full replica of the model + its own data slice.
- Forward and backward happen independently.
- When a bucket of gradients is computed (during backward), DDP does an all-reduce across all GPUs so every rank ends up with the averaged gradient.
- Optimizer.step is then identical across GPUs → weights stay in sync.
- Gradient bucketing overlaps communication with computation, hiding all-reduce latency.
Model parallel vs data parallel — when do you need each?
hard- Data parallel: replicate the model, split the batch across GPUs — dominant approach up to ~10B params on a single node.
- Model parallel: split the model itself across GPUs — needed when the model exceeds one GPU's memory.
- Two flavors: tensor parallel (split each matrix multiply across GPUs, e.g., Megatron) and pipeline parallel (put different layers on different GPUs, with microbatching to keep them all busy).
- Modern LLM training combines all three (3D parallelism).
Explain the three ZeRO stages.
hard- ZeRO (Zero Redundancy Optimizer) partitions training state across GPUs to save memory.
- Stage 1: shard the optimizer state (Adam's m and v) across N GPUs → memory / N.
- Stage 2: also shard the gradients.
- Stage 3 (FSDP): also shard the parameters themselves — each GPU only holds a 1/N slice of the model at rest and gathers full weights only when needed.
- Enables training LLMs many times bigger than a single-GPU capacity.
How is FSDP different from DDP?
hard- PyTorch FSDP (Fully Sharded Data Parallel) implements ZeRO-3: each GPU stores only a shard of the parameters, gradients, and optimizer state.
- When a layer runs forward, FSDP all-gathers its full parameters, computes, then discards them (or shards again).
- Enables 10-100x larger models than DDP on the same hardware, at the cost of extra communication.
- Modern default for training large models in PyTorch.
What is an EMA of weights and why do modern training recipes use it?
medium- Maintain a shadow copy of the weights θema updated as θema ← α⋅θema+(1−α)⋅θ every step, with α ~ 0.999-0.9999.
- Evaluate / deploy with θema, not the raw training weights.
- The EMA lives at the center of a flat minimum → ~0.5-1% better accuracy for free, better calibration, more stable predictions.
- Widely used in diffusion training (essential for sample quality), BYOL / DINO self-supervised methods (EMA teacher), and modern classification recipes.
State the Chinchilla scaling insight in one sentence.
hard- For a given compute budget C = 6 * N * T (params × tokens), the compute-optimal split has N and T roughly proportional (~20 tokens per parameter), meaning older large models (GPT-3, Gopher) were undertrained: same compute would have given a smaller model trained on more data with much better loss.
- Reshaped modern LLM training toward smaller-but-longer-trained models (LLaMA, Mistral).
Your validation loss is lower than your training loss. Is something broken?
medium- Usually not, and there are three ordinary explanations before you go looking for a bug.
- Regularization is active during training but not at evaluation, so dropout and stochastic depth make the training forward pass genuinely harder than the validation one.
- Augmentation does the same thing more strongly, since the model is scored on clean images but trained on distorted ones.
- And the reported training loss is an average over the epoch while the model was still improving, whereas validation is measured once at the end, so the training number reflects an older, worse model by roughly half an epoch.
- The signs that it really is a bug are a validation set that is easier than the training set because of a bad split, leakage of training examples into validation, or evaluation code that silently skips the hard cases.
Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?
hard- Batch norm estimates the mean and variance from the batch itself, so with four samples those statistics are extremely noisy.
- The noise acts as an uncontrolled regularizer during training and, worse, the running averages accumulated for inference no longer match what the layer saw, so train and eval behaviour diverge.
- It also couples examples within a batch, which breaks any assumption of per-sample independence.
- The practical fixes are to switch to group norm or layer norm, whose statistics are computed per sample and are therefore batch-size independent, or to use a normalization-free architecture with careful initialization.
- This is exactly why detection and segmentation models, which run tiny batches of large images, standardly use group norm.
Your augmentation pipeline made validation accuracy worse. What went wrong?
medium- Almost always a distribution mismatch: the augmentation created inputs unlike anything at test time, so the model spent capacity on a harder problem than the one it is graded on.
- Classic examples are horizontal flips on digits or text, aggressive colour jitter when colour is the label signal, and rotations on medical scans with a canonical orientation.
- Second possibility is that augmentation was accidentally applied to the validation set, which changes the measurement rather than the model.
- Third, the augmentation is fine but too strong for the training budget, since heavier augmentation needs more epochs to pay off.
- The diagnostic is to look at augmented samples with your own eyes and ask whether the label is still correct and the image still plausible.
How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?
medium- You run several smaller micro-batches, summing or averaging their gradients, and only step the optimizer once at the end.
- Mathematically the update matches a single large batch, so the loss curve is nearly identical while peak activation memory stays at the micro-batch level.
- What it does not fix is time: you still do the same amount of computation, so the step takes proportionally longer, and it does not give the throughput benefit of a genuinely larger batch on more hardware.
- It also interacts badly with batch normalization, whose statistics are computed per micro-batch and therefore reflect the small size, not the effective one.
- Remember to scale the loss so accumulation averages rather than sums.