EasyDeepLearn

How does INT8 weight quantization work in practice?

medium

Answer

  • For each weight tensor W (fp16, shape [out, in]), compute a per-channel scale  =  max⁡∣Wi\mathrm{scale}\; = \;\operatorname{max} \mid W_{i}| / 127 and quantized weight Wq  =  round(W  /  scale)W_{q}\; = \;\mathrm{round}(W\; / \;\mathrm{scale}) as int8.
  • At inference, compute WqW_{q} @ xfp16x_{\mathrm{fp16}} → dequantize output with scale.
  • Modern kernels (bitsandbytes, torch quant, TensorRT) fuse quantize + matmul + dequantize into a single INT8 tensor core op.
  • Memory: 2× less; latency: 1.5-2× faster on GPUs with INT8 tensor cores.
  • Accuracy loss: usually < 0.5% on quality benchmarks.
Check yourself — multiple choice
  • Global scale only
  • Per-channel scaling: Wq  =  round(W/scale)W_{q}\; = \;\mathrm{round}(W / \mathrm{scale}) as int8; fused kernels → 2× memory savings + 1.5-2× speedup + <0.5% quality loss
  • Only for training
  • Requires no scale

INT8 quant: per-channel scale + fused kernels → 2× memory, 1.5-2× speedup.

#quantization#inference

Practise LLMs & GenAI

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions