EasyDeepLearn
Deep Learning · section 19 of 19

Compression & interpretability

4 interview questions on compression & interpretability, each answered in full. Free to read, no account needed.

How does knowledge distillation work?

medium
  • Train a small 'student' network to match the outputs (usually soft probabilities at high temperature T) of a large pretrained 'teacher'.
  • Loss  =  α  ⋅  CE(student,  hardlabels)  +  (1−α)  ⋅  T2  ⋅  KL⁡(softmax⁡(student/T)  ∣∣  softmax⁡(teacher/T))\mathrm{Loss}\; = \;{\alpha}\; \cdot \;\mathrm{CE}(\mathrm{student}, \;\mathrm{hard}_{\mathrm{labels}})\; + \;(1 - {\alpha})\; \cdot \;T^{2}\; \cdot \;\operatorname{KL}(\operatorname{softmax}(\mathrm{student} / T)\; \mid \mid \;\operatorname{softmax}(\mathrm{teacher} / T)).
  • The soft targets carry more information than hard labels (they encode inter-class similarities), so the student often achieves 90-95% of the teacher's accuracy at 5-10x smaller size.
  • Foundational for model compression, mobile deployment, MLM speedups (DistilBERT).
#distillationPermalink & quiz →

What are the main pruning techniques for neural networks?

medium
  • Unstructured pruning: set individual weights to zero based on magnitude (∣w∣  <  threshold)( \mid w \mid \; < \;\mathrm{threshold}) — high compression, but sparse matrices are hard to accelerate without specialized kernels.
  • Structured pruning: remove entire filters / channels / attention heads — smaller matrices, natively fast on GPUs.
  • Typical recipe: iterative magnitude pruning + fine-tuning to recover accuracy.
  • Modern LLM pruning: SparseGPT, Wanda — one-shot pruning post-training.
  • Combined with quantization for further compression.
#pruning#distillationPermalink & quiz →

Post-training quantization vs Quantization-Aware Training — trade-offs?

hard
  • Post-training (PTQ): calibrate on a small dataset then quantize weights (and optionally activations) from fp32 to int8 / int4.
  • Fast (a few minutes), zero training overhead, some accuracy drop especially at ≤ 4 bits.
  • Quantization-aware training (QAT): simulate quantization noise during training (fake quant ops) so weights adapt — much smaller accuracy drop, especially at low bit widths, at the cost of a full fine-tuning run.
  • Popular libs: bitsandbytes, GPTQ, AWQ, LLM.int8().
#quantizationPermalink & quiz →

You need to halve inference cost. Do you quantize, prune, or distill?

medium
  • Quantize first, because it is the cheapest to try and the best understood.
  • Post-training quantization to 8-bit integers typically costs a fraction of a point of accuracy and needs only a small calibration set, and 4-bit weight-only quantization is now routine for language models.
  • Structured pruning gives real speedups only when the sparsity pattern matches what the hardware can exploit, since unstructured sparsity often saves memory without saving time.
  • Distillation gives the largest gains but is the most expensive, because it means training a new smaller model against the teacher's outputs, and it needs enough unlabelled data.
  • The pragmatic order is quantize, measure, then distil if you still need more, and prune only where the hardware rewards the pattern you can produce.
#quantization#pruning#distillationPermalink & quiz →

Practise Deep Learning

All 19 sectionsFlashcardsQuiz