TensorRT — what optimizations does it apply?
hardAnswer
- NVIDIA's inference optimizer for CUDA.
- (1) Layer fusion: combine consecutive ops (conv + bias + relu → single kernel).
- (2) Precision: fp16 / int8 / int4 quantization with calibration.
- (3) Kernel auto-tuning: pick best CUDA kernel per layer for target GPU.
- (4) Dynamic shapes support.
- (5) Multi-stream execution.
- Result: 2-10x speedup vs raw PyTorch on NVIDIA.
- TensorRT-LLM adds LLM-specific: paged KV, flash attention, in-flight batching.
- Standard for NVIDIA production.
Check yourself — multiple choice
- Random
- NVIDIA inference optimizer: layer fusion + fp16/int8/int4 quant with calibration + kernel auto-tuning per GPU + dynamic shapes + multi-stream; 2-10x speedup; TensorRT-LLM adds paged KV/flash attn/in-flight batching
- Just faster
- Not real
TensorRT: fusion + quant + kernel tuning; 2-10x; TRT-LLM for LLMs.
#serving#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.