What is FP8 inference and where does it help?
hardAnswer
- 8-bit floating-point (typically E4M3 or E5M2 formats) available on H100 / H200 / MI300 GPUs.
- Better dynamic range than INT8 (activations don't need per-channel scaling calibration), 2x faster than FP16 on tensor cores.
- Used for both weights and activations in modern serving stacks (TensorRT-LLM FP8, vLLM FP8-KV cache).
- Quality: near-zero loss vs FP16.
- FP8-KV cache alone halves KV memory → longer context or larger batch.
Check yourself — multiple choice
- Only INT quant available
- 8-bit float on H100+ (E4M3/E5M2); 2× faster than FP16, near-zero quality loss; FP8-KV cache halves KV memory
- Same as INT8
- Not supported anywhere
FP8: 8-bit float on H100+ → 2× faster than FP16, near-zero loss; halves KV cache.
#quantization#inference
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is quantization and what tradeoffs come with it?
- How does INT8 weight quantization work in practice?
- Compare GPTQ, AWQ, and NF4 quantization.
- How do you quantize the KV cache and why?
- What is the context window and what tricks extend it?
- What is the KV cache and why does it matter for LLM inference?