You need to halve inference cost. Do you quantize, prune, or distill?

medium

Answer

$Quantize first, because it is the cheapest to try and the best understood.$
$Post-training quantization to 8-bit integers typically costs a fraction of a point of accuracy and needs only a small calibration set, and 4-bit weight-only quantization is now routine for language models.$
$Structured pruning gives real speedups only when the sparsity pattern matches what the hardware can exploit, since unstructured sparsity often saves memory without saving time.$
$Distillation gives the largest gains but is the most expensive, because it means training a new smaller model against the teacher's outputs, and it needs enough unlabelled data.$
$The pragmatic order is quantize, measure, then distil if you still need more, and prune only where the hardware rewards the pattern you can produce.$

Check yourself — multiple choice

Prune first, it is free
Quantize first as the cheapest reliable win, then distil for larger gains, and prune only when the hardware can exploit the sparsity pattern
Distillation is always fastest to try
None of these reduce cost

Effort and reliability order the choice: quantization, then distillation, with pruning gated on hardware support.

#quantization#pruning#distillation

214 interview questions in this topic.