EasyDeepLearn

All 214 Deep Learning interview questions

Neural networks, training tricks, CNNs, RNNs, transformers and everything in between.

Every question below opens on its own page, with a full answer, the formulas written out and a multiple-choice check. Free to read, no account needed.

LessonFlashcardsQuizMock

Activations & fundamentals

  1. ReLU vs sigmoid vs GELU — when do you use each?easy
  2. In one sentence, what is backpropagation?easy
  3. State the universal approximation theorem in one sentence and its practical caveat.medium
  4. Why does stacking linear layers without nonlinearity collapse to a single linear layer?easy
  5. Depth vs width — which do you scale first?medium
  6. Why can't you initialize a neural net with all zeros?easy
  7. What is a computation graph and why does autograd care?easy
  8. What is 'dying ReLU' and how do you fix it?medium
  9. What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?medium
  10. What does temperature do in a softmax?easy
  11. Why compute softmax + cross-entropy jointly via log-sum-exp?medium
  12. How should the output head be designed for a regression task with a strictly positive target?medium
  13. How is the output head different for multi-label vs multi-class classification?easy
  14. Do you need bias terms in every layer of a modern neural network?medium
  15. How do you compute the parameter count of a linear layer and a convolution?medium
  16. How do FLOPs scale for a forward pass through an MLP and a transformer?hard
  17. Why does tanh cause vanishing gradients in deep nets?medium
  18. What is a forward hook in PyTorch and when is it useful?easy
  19. In one sentence, what is broadcasting and why does it matter?easy
  20. Compute the output shape of a 2D convolution with input (H, W), kernel k, stride s, padding p, dilation d.medium
  21. Write the vanilla RNN update equation and explain why it struggles with long sequences.easy
  22. MSE vs MAE for regression — when is each preferred?easy
  23. Why is 'overfit a single batch' the first test you should run on a new model?easy
  24. The same training script gives different results on two runs. How much of that can you remove?medium
  25. What do residual connections actually fix?medium

Backpropagation & autograd

  1. Why does the backward pass require more memory than the forward pass?medium
  2. What is backpropagation through time (BPTT)?easy

Initialization & gradients

  1. What are vanishing and exploding gradients, and how do you fix them?medium
  2. What is Xavier (Glorot) initialization and why?medium
  3. Why does He initialization use Var = 2 / fan_in instead of Xavier's 2 / (fan_in + fan_out)?medium
  4. When is orthogonal weight initialization useful?hard
  5. What causes exploding gradients and how do you handle them?medium
  6. Gradient clipping by norm vs by value — which do you prefer?easy

Training dynamics

  1. What does Batch Normalization do?medium
  2. How does dropout work and when is it applied?easy
  3. Why use a learning-rate schedule (warmup + cosine decay)?medium
  4. Why do residual (skip) connections help training deep networks?easy
  5. What is gradient accumulation and when do you use it?easy
  6. How does batch size affect training dynamics?medium
  7. What is deep double descent?hard
  8. What is grokking in deep learning?hard
  9. In one sentence, what is the Lottery Ticket Hypothesis?hard
  10. What is the 'implicit bias' of SGD?hard
  11. Your training loss becomes NaN. Debug it.medium
  12. What problem does LAMB solve?hard
  13. Describe the LR-range finder (Smith, 2015).easy
  14. What is the one-cycle policy?medium
  15. How should learning rate scale with batch size?medium
  16. What is gradient noise scale (GNS) and how do you use it?hard
  17. What is LARS and when is it used?hard
  18. Why does a model with BatchNorm behave differently at train time vs eval time?medium
  19. What is stochastic depth / DropPath and where is it used?medium
  20. What is early stopping and how do you configure it?easy
  21. What is Stochastic Weight Averaging (SWA)?medium
  22. How does snapshot ensembling work?medium
  23. What is adversarial training (PGD, FGSM)?hard
  24. How are input images typically normalized for pretrained CNNs?easy
  25. What is a solid default augmentation recipe for training a modern vision model?medium
  26. How should training resolution be chosen for a CNN or ViT?medium
  27. When do you use truncated BPTT and what's the trade-off?medium
  28. What is teacher forcing and its main pitfall?medium
  29. What is scheduled sampling?medium
  30. Contrast the pretraining objectives of BERT, GPT, and T5.medium
  31. What is deep supervision / auxiliary loss?medium
  32. How does mixed-precision training work with fp16 / bf16?medium
  33. Why does fp16 training need loss scaling?hard
  34. How does PyTorch DDP work under the hood?medium
  35. Model parallel vs data parallel — when do you need each?hard
  36. Explain the three ZeRO stages.hard
  37. How is FSDP different from DDP?hard
  38. What is an EMA of weights and why do modern training recipes use it?medium
  39. State the Chinchilla scaling insight in one sentence.hard
  40. Your validation loss is lower than your training loss. Is something broken?medium
  41. Your model trains well at batch size 256 but degrades at batch size 4. Why might batch normalization be the culprit?hard
  42. Your augmentation pipeline made validation accuracy worse. What went wrong?medium
  43. How does gradient accumulation let you train with a batch that does not fit in memory, and what does it not fix?medium

Optimizers

  1. SGD vs Adam vs AdamW — how do you choose?medium
  2. Write the vanilla SGD update rule and explain each term.easy
  3. How does classical momentum modify SGD?easy
  4. What is Nesterov momentum and how is it different from classical momentum?medium
  5. What is AdaGrad and its main drawback?medium
  6. How does RMSProp fix AdaGrad?medium
  7. Write Adam's full update rule.medium
  8. Why does AdamW work better than Adam + L2 weight decay for transformers?hard
  9. What is the Lion optimizer and why is it interesting?hard
  10. Why does Adafactor use less memory than Adam?hard
  11. What is Shampoo and when does second-order optimization pay off?hard
  12. How does weight decay change the objective and the update?medium
  13. How much extra memory does Adam use versus SGD?medium
  14. What does SAM (Sharpness-Aware Minimization) do?hard
  15. Why is weight decay via L2-in-loss different from decoupled weight decay in Adam?hard
  16. When do you prefer SGD with momentum over Adam?medium
  17. Why does AdamW handle weight decay differently from Adam, and why does it matter?hard

Learning-rate schedules

  1. What are cyclical learning rates (CLR)?medium
  2. Why do transformers need learning-rate warmup?medium
  3. What is cosine annealing and why is it preferred over step decay?medium
  4. When is step decay still appropriate?easy
  5. How does ReduceLROnPlateau work?easy
  6. What is SGDR (warm restarts) and when does it help?medium
  7. Why does transformer training usually need a learning-rate warmup?hard

Normalization

  1. When do you use LayerNorm instead of BatchNorm?medium
  2. What is GroupNorm and when do you use it?medium
  3. What is InstanceNorm and where does it shine?medium
  4. What is Weight Normalization and its trade-off?medium
  5. How does RMSNorm differ from LayerNorm?medium
  6. Pre-norm vs post-norm transformers — which is standard and why?hard
  7. What is Synchronized BatchNorm (SyncBN) and when do you need it?medium
  8. Why is BatchNorm awkward in RNNs?medium
  9. Why did transformers switch from post-norm to pre-norm at scale?hard

Regularization

  1. What is DropConnect and how is it different from Dropout?hard
  2. Where is dropout typically inserted in a transformer block?medium
  3. What is label smoothing and why does it help?easy
  4. How does Mixup work?medium
  5. What is Cutout / Random Erasing?easy
  6. Why is data augmentation effectively free regularization?easy
  7. What is consistency regularization?medium
  8. What is Manifold Mixup?hard
  9. Why does dropout play a smaller role in large language model training than in older vision models?hard

Data augmentation & mixing

  1. How does CutMix differ from Mixup?medium
  2. What is RandAugment and why is it a nice augmentation policy?medium
  3. What is Test-Time Augmentation (TTA)?easy

Convolutional networks

  1. What inductive biases do CNNs have?medium
  2. What is the receptive field in a CNN?medium
  3. SAME vs VALID vs REFLECT padding — what are the practical differences?easy
  4. What is a dilated (atrous) convolution?medium
  5. What is a transposed convolution ('deconv') and its checkerboard artifact issue?hard
  6. How does a depthwise separable convolution reduce compute?medium
  7. What are 1x1 convolutions used for?easy
  8. Max pooling vs average pooling — when do you pick each?easy
  9. Why does Global Average Pooling replace the FC head in modern CNNs?medium
  10. What is adaptive pooling and why is it useful?easy
  11. What is ConvNeXt's philosophy?hard

CNN architectures

  1. What made AlexNet (2012) a breakthrough?easy
  2. What is the design principle behind VGG?easy
  3. What is the Inception module?medium
  4. What is the key insight of ResNet's identity mapping?medium
  5. How is DenseNet different from ResNet?medium
  6. What is MobileNet designed for?medium
  7. What is compound scaling in EfficientNet?hard
  8. How does a Vision Transformer (ViT) treat images?hard
  9. What does Swin Transformer do differently from ViT?hard
  10. How does a two-stage detector (Faster R-CNN) work?hard
  11. How does YOLO / SSD / RetinaNet differ from two-stage detection?medium
  12. What is Non-Maximum Suppression (NMS) and its variants?medium
  13. What are anchor boxes and their downsides?medium
  14. Why is U-Net so popular for segmentation?medium
  15. How does Mask R-CNN extend Faster R-CNN for instance segmentation?hard
  16. What is a Feature Pyramid Network (FPN)?hard
  17. How does DETR reformulate object detection?hard
  18. Describe a single transformer encoder block in detail.hard
  19. How does a decoder block differ from an encoder block?hard
  20. Encoder-only vs decoder-only vs encoder-decoder — when do you pick each?medium
  21. Explain focal loss and why it's used in dense detection.hard
  22. Write Dice loss and its role in segmentation.medium
  23. What is perceptual loss and why is it better than pixel-wise for image quality?medium
  24. What are the main ideas of StyleGAN?hard
  25. How does Latent Diffusion (Stable Diffusion) reduce compute?hard
  26. What are normalizing flows and their trade-off vs GANs / diffusion?hard
  27. What is Neural Architecture Search (NAS)?hard

Object detection & segmentation

  1. What are IoU / Jaccard and Tversky losses?hard

Recurrent networks

  1. Why did transformers replace RNNs for sequence modeling?medium
  2. Explain the four gates of an LSTM.medium
  3. GRU vs LSTM — how are they different in practice?medium
  4. How does a bidirectional RNN work and when is it appropriate?easy
  5. What was Bahdanau attention and why was it a big deal?hard
  6. How does beam search work and what is its main failure mode?medium
  7. Why is length normalization needed in beam search?medium
  8. What is CTC loss and where is it used?hard

Attention & transformers

  1. How does self-attention work in a transformer?hard
  2. Why does a transformer need positional encoding?hard
  3. Why is attention scaled by 1/sqrt(d_k)?hard
  4. Why multi-head attention instead of a single big head?medium
  5. How do you pick the head dimension d_head?hard
  6. Why is standard self-attention O(n²) in sequence length?medium
  7. How do Performer / Linformer / linear attention reduce O(n²)?hard
  8. What is sparse / sliding-window attention?medium
  9. How do Longformer / BigBird combine sparse and global attention?hard
  10. What is Flash Attention?hard
  11. What is the KV cache in transformer inference?hard
  12. Explain sinusoidal positional encoding.medium
  13. What is the main limitation of learned absolute positional embeddings?medium
  14. How does relative positional encoding (Shaw / T5) work?hard
  15. How does ALiBi encode position?hard
  16. How does Rotary Position Embedding (RoPE) work?hard
  17. How does cross-attention differ from self-attention?medium
  18. How much can you interpret a model from attention weights?hard
  19. Briefly, what do BLEU and ROUGE measure?medium
  20. What are NTK-aware RoPE scaling and YaRN?hard
  21. How do you extend a transformer to very long contexts?hard
  22. Attention is quadratic in sequence length. Why is FlashAttention still a major win without changing that?hard

Losses for deep learning

  1. Write the Huber loss and explain when to use it.medium
  2. What is log-cosh loss and its advantage over Huber?medium
  3. Write binary cross-entropy for probability p̂ and label y ∈ {0, 1}.easy
  4. Multi-class cross-entropy vs one-vs-rest — pros and cons?medium
  5. Write the basic contrastive loss and when it's used.medium
  6. What is triplet loss and its main challenge?hard
  7. Write info-NCE / NT-Xent and its role in SSL.hard
  8. What is the standard reconstruction loss for autoencoders?easy
  9. Write the vanilla GAN's minimax objective.hard
  10. What does WGAN + gradient penalty change vs vanilla GAN?hard
  11. Write the VAE ELBO and explain each term.hard

Transfer learning & fine-tuning

  1. What is transfer learning and when is fine-tuning better than feature extraction?medium
  2. What is LoRA and why is it the standard for parameter-efficient fine-tuning?hard
  3. With a small labelled dataset and a large pretrained backbone, which parameters do you actually train?medium

Efficiency & distributed training

  1. What is gradient checkpointing and its trade-off?medium
  2. How does activation checkpointing interact with gradient accumulation and FSDP?hard

Generative models

  1. What is mode collapse in GANs and how do you mitigate it?hard
  2. Describe the forward and reverse processes in diffusion models.hard
  3. How does DDIM speed up diffusion sampling?hard
  4. What is classifier-free guidance (CFG) in diffusion?hard

Compression & interpretability

  1. How does knowledge distillation work?medium
  2. What are the main pruning techniques for neural networks?medium
  3. Post-training quantization vs Quantization-Aware Training — trade-offs?hard
  4. You need to halve inference cost. Do you quantize, prune, or distill?medium

Other topics