Compare GPTQ, AWQ, and NF4 quantization.
hardAnswer
- GPTQ (Frantar 2022): post-training layer-by-layer 4-bit weight quantization that minimizes reconstruction error via a Hessian-aware update.
- Precise, slow to compute (hours for 70B).
- AWQ (Lin 2023): 'Activation-aware Weight Quantization' — preserves the small fraction of weights corresponding to high-magnitude activations at higher precision.
- Faster than GPTQ, similar quality.
- NF4 (bitsandbytes / QLoRA): 4-bit 'NormalFloat' data type designed for normally-distributed weights → simple, works well as base for QLoRA fine-tuning.
- Practical: GPTQ/AWQ for pure serving, NF4 for QLoRA training.
Check yourself — multiple choice
- All identical
- GPTQ: Hessian-aware post-hoc quant; AWQ: preserve outlier weights; NF4: 4-bit dtype for weights, used in QLoRA training
- Only INT4 exists
- Only for classification
GPTQ / AWQ: serving quant; NF4: QLoRA training quant.
#quantization#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is quantization and what tradeoffs come with it?
- How does INT8 weight quantization work in practice?
- What is FP8 inference and where does it help?
- How do you quantize the KV cache and why?
- What is the context window and what tricks extend it?
- What is the KV cache and why does it matter for LLM inference?