25 interview questions on activations & fundamentals, each answered in full. Free to read, no account needed.
ReLU vs sigmoid vs GELU — when do you use each?
easy- Sigmoid/tanh saturate and cause vanishing gradients — avoid in hidden layers of deep nets, keep sigmoid for a binary output.
- ReLU is the default hidden activation: fast, sparse, but suffers from 'dying ReLU' (permanently zero neurons).
- Leaky ReLU / ELU fix that.
- GELU is a smooth approximation of ReLU and is the standard in transformers (BERT, GPT).
- SiLU/Swish is also popular in modern vision and LLMs.
In one sentence, what is backpropagation?
easy- Backpropagation is the reverse-mode automatic differentiation algorithm that efficiently computes gradients of a scalar loss with respect to every parameter of a neural network by traversing the computation graph backwards, applying the chain rule at each node.
State the universal approximation theorem in one sentence and its practical caveat.
medium- A feedforward network with a single hidden layer of finite width can approximate any continuous function on a compact domain to arbitrary accuracy — but the theorem says nothing about width required, training feasibility, or generalization.
- In practice depth is much more parameter-efficient than width, which is why deep networks dominate.
Why does stacking linear layers without nonlinearity collapse to a single linear layer?
easy- The composition of linear maps is a linear map: W2 (W1x+b1)+b2=(W2W1) x+(W2b1+b2).
- No matter how many layers you stack, the effective function is W x + b.
- Non-linear activations (ReLU, GELU, tanh) break this and unlock representational power that grows with depth.
Depth vs width — which do you scale first?
medium- Depth compounds representational capacity exponentially in the number of piecewise-linear regions for ReLU networks, while width only grows it polynomially.
- So depth is usually the better initial investment for representation.
- However, very deep networks are harder to optimize — need normalization, residual connections, and careful init.
- Modern architectures (transformers, ConvNeXt) balance both via 'compound scaling'.
Why can't you initialize a neural net with all zeros?
easy- Symmetric weights mean every neuron in a layer computes the same output and receives the same gradient — the layer effectively has one neuron.
- Training never breaks the symmetry.
- Break it with any random init (Xavier / He / uniform).
- Biases can be initialized to zero because they aren't multiplied together; only the weight matrices need randomness.
What is a computation graph and why does autograd care?
easy- A DAG whose nodes are operations and edges track data + gradient dependencies.
- During the forward pass, each op records its inputs so its backward function can be called.
- Reverse-mode autodiff (backprop) traverses the graph from the loss backwards, applying the chain rule at each node.
- PyTorch builds it dynamically per iteration; TensorFlow (TF1) and JAX build it statically.
What is 'dying ReLU' and how do you fix it?
medium- A ReLU neuron that always outputs zero has a zero gradient everywhere and stops learning permanently.
- Causes: large negative bias, extreme learning rate, poor init.
- Fixes: Leaky ReLU or ELU (nonzero slope on the negative side), He initialization, lower learning rate, better data normalization, or use GELU / SiLU which are smooth and don't fully saturate to zero.
What are SiLU (Swish) and GELU, and why do transformers prefer them over ReLU?
medium- SiLU(x)=x⋅sigmoid(x).
- GELU(x) ≈ x * Phi(x) — smooth version of ReLU using a Gaussian CDF.
- Both are smooth and non-monotonic near zero, giving richer gradients than ReLU's hard elbow.
- Empirically better on transformers and modern vision (ConvNeXt, EfficientNet).
- GELU is standard in BERT / GPT / most LLMs; SiLU in vision (EfficientNet) and some LLMs (Llama, Mistral).
What does temperature do in a softmax?
easy- softmax(z/T): T > 1 flattens the distribution (softer, higher entropy — used in distillation for teacher outputs), T < 1 sharpens toward one-hot (used in decoding to make choices more deterministic), T = 0 is argmax.
- In LLM decoding, temperature is a knob for creativity vs determinism.
- In distillation, teacher and student use matching high T so students learn full distribution info.
Why compute softmax + cross-entropy jointly via log-sum-exp?
medium- Naive softmax=exp(zi)/sum(exp(zj)) overflows for large logits.
- Compute log−softmax(zi)=zi−logsumexp(z) where logsumexp(z) = max(z)+log(sum(exp(z−max(z)))).
- Numerically stable.
- Cross-entropy then becomes -sum(target * log-softmax) — computed in one op (nn.CrossEntropyLoss in PyTorch), avoiding a separate softmax that could overflow before the log.
How should the output head be designed for a regression task with a strictly positive target?
medium- Options: (1) predict log(y) and exponentiate — implicit positivity, natural for right-skewed targets; (2) apply exp() or softplus at the output to guarantee positivity; (3) predict a mean of a Gamma/log-normal distribution via a GLM-style head.
- Never use identity output with MSE if y can only be positive — the model will predict negatives during training which then contribute noise.
How is the output head different for multi-label vs multi-class classification?
easy- Multi-class: one softmax head, one loss (categorical cross-entropy) — labels are mutually exclusive.
- Multi-label: K independent sigmoid heads, one binary cross-entropy loss per label, summed — a document can have multiple tags simultaneously.
- Do not use softmax for multi-label: it forces the K probabilities to sum to 1, which contradicts independence.
Do you need bias terms in every layer of a modern neural network?
medium- Often no.
- When a layer is followed by BatchNorm or LayerNorm with learned shift (beta), the bias is redundant — the norm's beta term absorbs it.
- Standard practice: linear + norm → set bias=False on the linear.
- Modern transformers frequently omit biases from Q/K/V projections and MLP layers when they use RMSNorm or LayerNorm — small memory / compute savings, no accuracy hit.
How do you compute the parameter count of a linear layer and a convolution?
medium- Linear(in, out): in * out + out (weights + bias).
- Conv2d(Cin,Cout,kernel=k,stride=s,pad=p): Cin⋅Cout⋅k⋅k+Cout.
- Multi-head attention with hidden d and n heads: 4⋅d2 (Q, K, V, out projections) + biases.
- MLP block: 2⋅d⋅dffn (up + down projection).
- Practical parameter count for a transformer layer ≈ 12⋅d2 (attn+MLPwithdffn=4d).
How do FLOPs scale for a forward pass through an MLP and a transformer?
hard- MLP layer(in, out) on batch B: 2 * B * in * out FLOPs.
- Transformer forward (per token, per layer) ≈ 2⋅nlayers⋅d⋅dffn (MLP)+2⋅nlayers⋅d⋅(d+dhead) (attentionlinearprojections)+2⋅nlayers⋅seq⋅d (attention softmax + matmuls).
- For long context, attention dominates because it's O(seq2⋅d).
- Approximation: forward FLOPs ≈ 2 * N (params) per token — thus training a model with N params on T tokens takes ~6 * N * T FLOPs (forward + backward + optimizer).
Why does tanh cause vanishing gradients in deep nets?
medium- tanh'(x)=1−tanh2(x) ≤ 1, and is close to 1 only near x = 0.
- Deep nets keep multiplying such derivatives during backprop — the product shrinks toward 0.
- With He/Xavier init, activations at each layer land in the saturating region (∣x∣>2), where the derivative is near zero.
- Fix: use ReLU/GELU which don't saturate on the positive side, plus normalization to keep activations in the useful range.
What is a forward hook in PyTorch and when is it useful?
easy- A callback registered on a module that fires during the forward pass, receiving (module, input, output).
- Useful for extracting intermediate features (e.g., pen-ultimate embeddings for downstream models), debugging shape / value issues, visualizing activations, running feature-map probes, or computing custom regularizers on hidden activations without modifying the model code.
- Backward hooks exist too — for inspecting gradients per module.
In one sentence, what is broadcasting and why does it matter?
easy- Broadcasting extends tensor operations across dimensions of unequal shape by implicit repetition, with rules: align shapes from the right, dimensions of size 1 are broadcast to match, otherwise dimensions must match.
- It lets you write add, multiply, and mask operations across batches and features without explicit tile/repeat — critical for efficient GPU code.
Compute the output shape of a 2D convolution with input (H, W), kernel k, stride s, padding p, dilation d.
medium- Hout=floor((H + 2*p - d*(k-1) - 1) / s) + 1, same for W.
- Special cases: 'same' padding for stride 1 uses p = (k-1)/2 * d.
- Standard kernels are 3×3 with p=1, s=1 (preserves spatial size).
- Downsampling: stride 2 halves the resolution.
- Understanding this formula is a bread-and-butter interview requirement — several bug fixes in production code hinge on it.
Write the vanilla RNN update equation and explain why it struggles with long sequences.
easy- ht=tanh(Whh⋅ht−1+Wxh⋅xt+b).
- Repeatedly multiplying by Whh during BPTT causes gradients to vanish (spectral norm < 1) or explode (> 1) over long sequences.
- LSTMs / GRUs introduce gated additive updates to preserve gradients across time; transformers skip recurrence entirely and use attention.
MSE vs MAE for regression — when is each preferred?
easy- MSE (L2): differentiable everywhere, penalizes large errors quadratically → sensitive to outliers, gradients are proportional to error size.
- MAE (L1): penalizes linearly → robust to outliers but not differentiable at 0.
- Choose MSE when errors are Gaussian-ish and outliers are rare; choose MAE when the target has heavy tails or you want the median rather than the mean.
- Huber loss is the practical compromise.
Why is 'overfit a single batch' the first test you should run on a new model?
easy- Because it isolates implementation bugs from learning problems.
- A correctly wired model with enough capacity can drive the loss on a handful of examples to nearly zero, since it can simply memorize them.
- If it cannot, the fault is mechanical rather than statistical: labels misaligned with inputs, the loss reading the wrong axis, gradients not flowing because of a detached tensor or a frozen parameter, or a learning rate so small nothing moves.
- The test costs seconds and rules out an entire class of silent failures before you spend GPU hours.
- Only once a single batch overfits does it make sense to talk about regularization, augmentation, or architecture.
The same training script gives different results on two runs. How much of that can you remove?
medium- Less than people expect, and it is worth knowing which sources you control.
- Seeding the framework, Python and NumPy fixes weight initialization, dropout masks and shuffling order, which removes most of the variance.
- What remains comes from the hardware: several GPU kernels, notably atomics used in scatter operations and some convolution algorithms, accumulate in nondeterministic order, and floating-point addition is not associative, so results differ in the last bits and then diverge over thousands of steps.
- Frameworks expose a deterministic mode that swaps in slower ordered kernels, which gets you bit-identical runs at a real throughput cost.
- Data loader worker count and any non-seeded augmentation also matter.
- The practical stance is to seed everything, report results over several seeds rather than one, and reserve full determinism for debugging.
What do residual connections actually fix?
medium- They make optimization tractable at depth.
- Without them, a deep stack must learn an identity mapping through many nonlinear layers just to preserve information, which gradient descent does badly, and the observed symptom is that a deeper network trains to a worse training error than a shallower one.
- That is an optimization failure, not overfitting.
- A residual branch gives the gradient a path that reaches early layers with its magnitude roughly intact, so signal neither vanishes nor is distorted by the product of many Jacobians.
- The broader consequence is that the network can represent a shallow function easily and add depth only where it helps, which is why residual blocks are in essentially every modern architecture.