Gradient clipping by norm vs by value — which do you prefer?
easyAnswer
- Clip-by-norm: rescale the whole gradient vector so its L2 norm is at most .
- Preserves direction, only shrinks magnitude.
- Preferred default.
- Clip-by-value: clamp each element to [-clip, clip].
- Distorts the gradient direction — element-wise clipping can flip the relative magnitudes of components.
- Use clip-by-norm for RNNs and large-model training; clip-by-value only for specific numerical stability issues.
Check yourself — multiple choice
- Value clipping is always better
- Clip-by-norm preserves direction (preferred); clip-by-value distorts the gradient vector
- They're identical
- Never clip gradients
Norm clipping preserves direction; value clipping distorts it.
#gradients#training
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What are vanishing and exploding gradients, and how do you fix them?
- What causes exploding gradients and how do you handle them?
- What does Batch Normalization do?
- How does dropout work and when is it applied?
- Why use a learning-rate schedule (warmup + cosine decay)?
- Why do residual (skip) connections help training deep networks?