What is a forward hook in PyTorch and when is it useful?
easyAnswer
- A callback registered on a module that fires during the forward pass, receiving (module, input, output).
- Useful for extracting intermediate features (e.g., pen-ultimate embeddings for downstream models), debugging shape / value issues, visualizing activations, running feature-map probes, or computing custom regularizers on hidden activations without modifying the model code.
- Backward hooks exist too — for inspecting gradients per module.
Check yourself — multiple choice
- Hooks are only for backward passes
- A callback that fires during forward pass — extract intermediate features / debug / probe
- Hooks replace loss functions
- Hooks work only in TensorFlow
Forward hook = callback on module forward — great for extracting features and debugging.
#fundamentals
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- Depth vs width — which do you scale first?
- Why can't you initialize a neural net with all zeros?