What is quantization and what tradeoffs come with it?
mediumAnswer
- Quantization stores weights (and sometimes activations) in lower precision — INT8, INT4, or even lower — reducing memory and speeding up inference.
- Post-training quantization is cheap but can hurt quality; quantization-aware training keeps accuracy closer to FP16. 4-bit weight-only quantization (e.g., GPTQ, AWQ, bitsandbytes NF4) is the sweet spot for running big LLMs on smaller GPUs.
Check yourself — multiple choice
- Quantization always increases memory
- 4-bit weight-only quantization is a common tradeoff for large LLMs
- Quantization has no impact on accuracy
- Quantization-aware training is cheaper than post-training quantization
4-bit weight quantization keeps most of the quality and shrinks memory ~4x.
#quantization#compression#inference
Practise LLMs & GenAI
214 interview questions in this topic.