Post-training quantization vs Quantization-Aware Training — trade-offs?
hardAnswer
- Post-training (PTQ): calibrate on a small dataset then quantize weights (and optionally activations) from fp32 to int8 / int4.
- Fast (a few minutes), zero training overhead, some accuracy drop especially at ≤ 4 bits.
- Quantization-aware training (QAT): simulate quantization noise during training (fake quant ops) so weights adapt — much smaller accuracy drop, especially at low bit widths, at the cost of a full fine-tuning run.
- Popular libs: bitsandbytes, GPTQ, AWQ, LLM.int8().
Check yourself — multiple choice
- Same accuracy
- PTQ: fast, some drop especially <4-bit; QAT: fine-tune with fake quant ops → less drop, full training cost
- Only PTQ exists
- Quantization destroys models
PTQ: fast + some drop; QAT: fine-tune with fake quant → minimal drop at low bit width.
#quantization
Practise Deep Learning
214 interview questions in this topic.