How does quantization affect serving?

hard

Answer

$fp32 \to fp16 / bf16: 2x memory + speed, minimal accuracy loss. fp16 \to int8: 4x from fp32; needs calibration on representative data to compute per-channel scales; usually <1% accuracy loss. int4 / int2: aggressive, needs advanced methods (GPTQ, AWQ, SmoothQuant); for LLMs on consumer GPU.$
$FP8: newer NVIDIA H100 native.$
$Post-training vs QAT: QAT recovers more accuracy but requires retraining.$
$Standard for production LLMs (7B model in int4 \to 4GB VRAM).$

Check yourself — multiple choice

Random
fp32→fp16/bf16 2x mem/speed minimal loss; int8 4x from fp32 needs calibration <1% loss; int4/int2 needs GPTQ/AWQ/SmoothQuant; FP8 H100 native; QAT>PTQ for accuracy; 7B in int4 = 4GB VRAM
Just cast
Not real

Quant: fp16 2x, int8 4x, int4/2 aggressive; PTQ vs QAT.

#serving#advanced

215 interview questions in this topic.