What are the main pruning techniques for neural networks?
mediumAnswer
- Unstructured pruning: set individual weights to zero based on magnitude — high compression, but sparse matrices are hard to accelerate without specialized kernels.
- Structured pruning: remove entire filters / channels / attention heads — smaller matrices, natively fast on GPUs.
- Typical recipe: iterative magnitude pruning + fine-tuning to recover accuracy.
- Modern LLM pruning: SparseGPT, Wanda — one-shot pruning post-training.
- Combined with quantization for further compression.
Check yourself — multiple choice
- Only random deletion
- Unstructured (magnitude, hard to accelerate) vs structured (filters/channels/heads, natively fast); iterative or one-shot
- Pruning always hurts accuracy
- Same as quantization
Pruning: unstructured (sparse) vs structured (channel/head); combine with fine-tuning to preserve accuracy.
#pruning#distillation
Practise Deep Learning
214 interview questions in this topic.
Related questions
- You need to halve inference cost. Do you quantize, prune, or distill?
- What does temperature do in a softmax?
- In one sentence, what is the Lottery Ticket Hypothesis?
- What is an EMA of weights and why do modern training recipes use it?
- How does knowledge distillation work?
- What is LoRA and why is it the standard for parameter-efficient fine-tuning?