34 interview questions on infrastructure & serving, each answered in full. Free to read, no account needed.
Ray for ML — what does it provide?
medium- Distributed Python framework.
- (1) Ray Core: distributed tasks + actors.
- (2) Ray Train: distributed training (PyTorch / TF wrappers).
- (3) Ray Tune: hyperparameter search.
- (4) Ray Data: distributed data loading.
- (5) Ray Serve: model serving.
- (6) RLlib: distributed RL.
- One framework covering training → tuning → serving.
- Common on Kubernetes via KubeRay.
- Alternatives: Dask (arrays / DataFrames), Spark (batch + SQL).
Iceberg vs Delta Lake — which to pick?
hard- Both provide ACID over object storage.
- Iceberg: open-standard, engine-agnostic (Spark, Trino, Flink, Presto, Snowflake, Doris).
- Better hidden partitioning + evolution.
- Netflix / Apple / Adobe.
- Delta: Databricks-first, more optimizations in Databricks Runtime.
- OSS via Delta.io.
- Newer format Delta 3.0 = Uniform (read Delta + Iceberg).
- Pick Iceberg for multi-engine data platform; Delta for Databricks-centric.
Kubeflow — what does it provide?
medium- Kubernetes-native ML platform.
- Components: (1) Kubeflow Pipelines (KFP): DAG orchestration on K8s.
- (2) Katib: hyperparameter tuning + NAS.
- (3) KServe (formerly KFServing): model serving with autoscaling.
- (4) Notebooks: managed Jupyter on K8s.
- (5) Distributed training operators (PyTorch, TF, MPI, XGBoost).
- Modular: use pieces separately.
- Standard for K8s-first ML organizations.
- Alternatives: MLRun, ZenML, Metaflow.
NVIDIA Triton Inference Server — what makes it fast?
medium- (1) Multi-framework: TensorRT, ONNX, PyTorch, TensorFlow, custom Python backend.
- (2) Dynamic batching: request coalescing without user awareness.
- (3) Concurrent model execution on same GPU (model parallelism).
- (4) Model ensembling: pipeline models server-side.
- (5) HTTP + gRPC.
- (6) Metrics for Prometheus.
- (7) Model repository: hot-reload versions.
- Backed by NVIDIA + open-source.
- Standard for GPU serving at scale.
vLLM — why is it fast for LLM serving?
hard- (1) PagedAttention: virtual memory for KV cache (page-based), eliminates fragmentation → 2-4x throughput.
- (2) Continuous batching: at each step, start new sequences + finish completed ones (no waiting for batch to complete).
- (3) Optimized CUDA kernels.
- (4) Speculative decoding integration.
- (5) Quantization (AWQ, GPTQ, FP8).
- (6) Multi-GPU tensor parallelism.
- UC Berkeley project, now industry standard.
- Alternatives: TGI (HF), TensorRT-LLM (NVIDIA), SGLang, LMDeploy.
PagedAttention — what problem does it solve?
hard- Standard KV cache reserves contiguous memory per sequence for max length → 60-80% waste (short sequences reserve too much).
- Fragmentation prevents new sequences even with free memory.
- Paged attention: divide KV cache into fixed-size blocks (pages) similar to OS virtual memory.
- Sequence maps to non-contiguous pages via block table.
- Utilization → 90%+; throughput 2-4x.
- Enables efficient shared prefix caching (system prompt shared across users).
Continuous batching vs static batching.
hard- Static batching: form batch of N requests, run to completion (wait for slowest), then next batch.
- Problem: mix of long + short sequences wastes GPU on completed sequences.
- Continuous batching: at each token step, remove completed sequences + add new pending sequences.
- GPU fully utilized. 2-4x throughput improvement.
- Requires flexible KV cache (see PagedAttention).
- Industry standard: vLLM, TGI, TensorRT-LLM.
TorchServe — when to use it?
medium- Meta's PyTorch model server.
- Features: (1) built-in HTTP/gRPC API.
- (2) model archiver (.mar files).
- (3) multi-model + versioning.
- (4) batch inference.
- (5) metrics for Prometheus.
- (6) A/B routing via traffic split.
- Simpler than Triton, PyTorch-focused.
- Better for PyTorch-only teams without GPU optimization.
- Downside: less performant than Triton for GPU-heavy workloads.
- Losing mindshare to vLLM (LLM) + Triton (general).
TensorFlow Serving — when to use it?
medium- Google's TF model server.
- Features: (1) SavedModel format.
- (2) multi-model + versioning with staged rollout.
- (3) batching via config.
- (4) HTTP/REST + gRPC.
- (5) mature + battle-tested at Google.
- Downside: TF-only; less flexibility than Triton; declining share as PyTorch dominates.
- Still relevant for TF-based large orgs.
- Alternative: TF Serving in Triton (TF backend).
BentoML — what does it add?
medium- Python-first ML serving framework.
- Package model + preprocessing + business logic into 'Bento' (deployable unit).
- Features: (1) unified API across frameworks.
- (2) adaptive batching.
- (3) Docker image builder.
- (4) deploy to K8s / Serverless / EC2.
- (5) built-in monitoring.
- Better DX than raw Triton for Python-heavy pipelines.
- Modern choice for smaller teams.
- Downside: less optimized than Triton for pure GPU throughput.
KServe (KFServing) — what does it provide?
medium- Kubernetes-native model serving.
- Features: (1) autoscaling to zero (serverless).
- (2) canary + traffic splitting.
- (3) explainer / transformer sidecars.
- (4) multi-framework via serving runtimes (Triton, TorchServe, ONNX).
- (5) built-in metrics + tracing.
- Built on Knative.
- Standard for K8s-first orgs.
- Alternatives: Seldon Core (feature-richer), Ray Serve, custom Deployment + Service.
Serverless ML inference — pros / cons?
medium- AWS Lambda / Cloud Run / Cloud Functions run model on demand.
- Pros: (1) pay per invocation (great for spiky / low traffic).
- (2) auto-scale to zero.
- (3) no infra to manage.
- Cons: (1) cold start (100ms-30s depending on model size + runtime).
- (2) memory / duration limits (Lambda 10 GB / 15 min).
- (3) no GPU on most (Lambda has limited GPU).
- (4) more expensive per prediction at high traffic.
- Rule: <10 QPS + latency-tolerant → serverless; else provisioned.
How do you mitigate serverless cold-start latency?
medium- (1) Provisioned concurrency: pre-warmed containers (Lambda).
- (2) Snap start: pre-load runtime state (Java / .NET Lambda).
- (3) Reduce model size (quantize, distill).
- (4) Load model lazily on first request → keep warm via periodic ping (
warmer pattern). - (5) Container reuse: Lambda reuses container for successive invocations.
- (6) Move heavy imports to global scope (loaded once).
- (7) Consider always-on service if cold start unacceptable.
- Modern: Cloud Run min-instances = 1.
ONNX — why use it?
medium- Open Neural Network Exchange: standard format for cross-framework interop.
- Train in PyTorch → export ONNX → run in ONNX Runtime (C++, C#, Java, JS).
- Benefits: (1) framework-agnostic deployment.
- (2) ONNX Runtime is optimized (better than raw PyTorch inference).
- (3) edge deployment (mobile, embedded).
- (4) hardware-specific execution providers (CUDA, TensorRT, OpenVINO, DirectML).
- Downside: not all ops supported; custom ops require conversion.
- Standard for edge / cross-platform.
TensorRT — what optimizations does it apply?
hard- NVIDIA's inference optimizer for CUDA.
- (1) Layer fusion: combine consecutive ops (conv + bias + relu → single kernel).
- (2) Precision: fp16 / int8 / int4 quantization with calibration.
- (3) Kernel auto-tuning: pick best CUDA kernel per layer for target GPU.
- (4) Dynamic shapes support.
- (5) Multi-stream execution.
- Result: 2-10x speedup vs raw PyTorch on NVIDIA.
- TensorRT-LLM adds LLM-specific: paged KV, flash attention, in-flight batching.
- Standard for NVIDIA production.
How does quantization affect serving?
hard- fp32 → fp16 / bf16: 2x memory + speed, minimal accuracy loss. fp16 → int8: 4x from fp32; needs calibration on representative data to compute per-channel scales; usually <1% accuracy loss. int4 / int2: aggressive, needs advanced methods (GPTQ, AWQ, SmoothQuant); for LLMs on consumer GPU.
- FP8: newer NVIDIA H100 native.
- Post-training vs QAT: QAT recovers more accuracy but requires retraining.
- Standard for production LLMs (7B model in int4 → 4GB VRAM).
GPTQ vs AWQ vs SmoothQuant — key differences.
hard- GPTQ (Frantar): layer-by-layer quantization with second-order approximation; fast + good accuracy for LLMs. AWQ (Lin): identifies salient weights via activation magnitudes; protects them from quantization → better preservation of critical channels.
- SmoothQuant (Xiao): migrates quantization difficulty from activations to weights via smoothing → makes activation-quantization tractable.
- Modern LLM serving mostly uses AWQ (best quality) + GPTQ + smoothquant depending on hardware.
- Runtime: vLLM / TensorRT-LLM support all.
FlashAttention — why is it faster?
hard- Standard attention: O(N2) memory for QKT matrix.
- FlashAttention (Dao 2022): (1) tiling: compute attention block-by-block in SRAM (fast on-chip memory).
- (2) online softmax: recompute normalizer per tile.
- (3) never materialize full attention matrix. → 2-4x speedup + linear memory.
- FlashAttention 2 + 3 further optimize.
- Enables long-context LLMs (100k+ tokens).
- Adopted in all modern implementations (PyTorch native, vLLM, HF, xformers).
Speculative decoding — how does it accelerate LLMs?
hard- Small 'draft' model generates K tokens fast.
- Large 'target' model verifies all K in single forward pass (parallel).
- Accepted tokens (matching what large would have generated) are kept; on first mismatch, use large model's token + restart.
- Amortizes large-model forward pass across accepted tokens. 2-3x speedup with correct output distribution (matches large model exactly).
- Variants: Medusa (multi-head), Eagle, TriForce.
- Modern standard for LLM serving speedup.
Tensor parallelism vs pipeline parallelism vs data parallelism.
hard- Data parallelism (DDP / FSDP): same model replicated; different data per GPU; all-reduce gradients.
- Tensor parallelism (Megatron): split large layers (attention, FFN) across GPUs; each does portion of matmul + all-reduce.
- Pipeline parallelism (GPipe / PipeDream): different layers on different GPUs; micro-batches flow through pipeline.
- Combine all three for LLM training (3D parallelism).
- Modern: FSDP + TP + PP + expert parallelism (MoE).
- ZeRO stages 1-3 optimize memory.
FSDP / ZeRO — how do they save memory?
hard- Standard DDP: each GPU holds full model + gradients + optimizer state.
- Adam optimizer states = 2x model params.
- FSDP (PyTorch) / ZeRO-3 (DeepSpeed): shard model params + gradients + optimizer states across GPUs.
- All-gather params before forward, discard after.
- All-reduce gradients + reshard.
- Memory per GPU: (params+grads+optimizer)/Ngpus.
- Enables 10-100x larger models.
- ZeRO stages: 1 (shard optimizer), 2 (+gradients), 3 (+params).
- Standard for LLM training.
How do you choose batch size for inference?
medium- Trade-off: larger batch → higher throughput + higher latency.
- Constraints: (1) memory (KV cache scales with batch × seq).
- (2) latency SLA (p99 must fit).
- (3) GPU compute pattern (matmul kernels want ~64+ batch).
- Dynamic batching: aggregate requests over 5-50ms window until batch full or timeout.
- Modern LLM serving: continuous batching adjusts dynamically.
- Rule: profile QPS vs latency, pick knee of curve; often batch=8-32 sweet spot for online.
How do you use spot / preemptible instances safely for ML?
medium- 50-90% discount but can be reclaimed 2 min notice.
- Safe for: (1) training (checkpoint frequently, resume).
- (2) batch inference (retry).
- (3) experiment sweeps.
- Unsafe for online serving unless mixed with on-demand.
- Patterns: (1) checkpoint every N steps → resume on interruption.
- (2) mixed instance groups: on-demand baseline + spot burst.
- (3) spot fleet across zones + instance types → diversity reduces simultaneous reclaim.
- (4) SIGTERM handler for graceful shutdown.
- Save 50-80% compute.
How do you deploy ML to edge / mobile?
medium- (1) Model conversion: TFLite (Android), CoreML (iOS), ONNX Runtime Mobile, PyTorch Mobile.
- (2) Quantization aggressive: int8 / int4 for size.
- (3) Pruning + distillation.
- (4) Hardware-specific: NNAPI, Metal, DSP, NPU (Qualcomm Hexagon, Apple Neural Engine).
- (5) On-device fine-tuning (federated learning).
- (6) Model size 5-50 MB typical.
- (7) Offline capability.
- (8) Careful with battery / thermal.
- Use cases: face detection, keyboard suggestion, translation.
- Cloud fallback for hard cases.
Running ML in the browser — WebAssembly / WebGPU.
medium- Options: (1) TensorFlow.js: JS/WebGL/WebGPU backend; ONNX Runtime Web (WASM + WebGL / WebGPU).
- (2) Transformers.js: run HuggingFace models in browser.
- (3) WebLLM (MLC.ai): run LLM entirely in-browser via WebGPU.
- Advantages: (1) no server cost.
- (2) privacy (data never leaves device).
- (3) offline capability.
- (4) instant deploy.
- Disadvantages: (1) model download size (users notice 100+ MB).
- (2) limited GPU/memory (mobile).
- Use cases: privacy-first ML, PWAs, demos.
Which GPU for training vs inference?
medium- Training: (1) NVIDIA A100 / H100 (data center): 80GB VRAM, NVLink, HBM3, best.
- (2) H200 / B200 latest.
- (3) TPU v4/v5 (Google) alternative.
- (4) Rent from cloud (AWS p4/p5, GCP TPU, Lambda Labs, CoreWeave).
- Inference: (1) L40S / A10G (data center inference-optimized, cheaper).
- (2) T4 (older, budget).
- (3) L4 for video / low-power.
- (4) Consumer 4090 / 5090 for small teams (48GB via 4090 pairs).
- LLM inference: 7B fits on 1x A10G, 70B needs 2x A100 or 4x A10G with quant.
Network latency in ML serving — how much matters?
medium- Total latency = client → LB → service → feature fetch → model → response.
- Model inference often only 30-50% of total.
- Preprocess, feature store fetch, network hops dominate.
- Optimize: (1) co-locate feature store with model (same AZ / region).
- (2) precompute + cache features hot path.
- (3) HTTP/2 or gRPC keep-alive (reduce connection setup).
- (4) Nagle's off+TCPNODELAY for small payloads.
- (5) Client-side connection pooling.
- (6) Serverless has extra hop cost.
- Profile every hop.
Kubernetes for ML — key patterns.
medium- (1) Deployment + Service for online serving.
- (2) Job / CronJob for batch training / scoring.
- (3) StatefulSet for stateful (feature stores).
- (4) HPA for autoscaling.
- (5) NodePool with GPU labels + taints (tolerations for GPU workloads).
- (6) Volumes for shared data (S3-CSI, FSx, NFS).
- (7) Priority + preemption for expensive jobs.
- (8) Namespaces for isolation.
- Operators: Kubeflow, Ray, Volcano scheduler for batch.
- Cost tools: Kubecost.
- Standard for enterprise ML.
How can multiple models share a GPU?
hard- (1) MPS (Multi-Process Service, NVIDIA): concurrent process access, no isolation.
- (2) MIG (Multi-Instance GPU, A100+): hardware partition into 1-7 slices with dedicated compute + memory.
- (3) Triton concurrent model execution: single process, multiple models on GPU.
- (4) K8s device plugin: dev-plugin variants for fractional GPU (nvidia-device-plugin + MIG or MPS).
- (5) Time slicing: rare, high overhead.
- Rule: MIG for isolation / SLA, MPS for max utilization, Triton for cohabiting models.
How do you cache LLM prompts effectively?
hard- (1) KV cache reuse (prefix caching): shared system prompt cached across users; new requests reuse. vLLM / Anthropic support natively.
- Saves 30-90% compute for shared prefixes.
- (2) Result cache (memoize): hash prompt → cached response.
- Works for deterministic prompts.
- (3) Embedding cache: dedupe similar prompts via vector similarity + return cached.
- (4) Batch level: within batch, common prefix computed once.
- Modern: Anthropic 'prompt caching' + Gemini 'context caching' + OpenAI 'prompt caching' (auto).
Vector DB — which one and why?
medium- (1) Pinecone: SaaS, ease of use, expensive at scale.
- (2) Weaviate: OSS + SaaS, GraphQL, built-in vectorizer.
- (3) Qdrant: OSS Rust, high performance, self-host friendly.
- (4) Milvus: OSS, most scale-proven, complex ops.
- (5) pgvector: Postgres extension, simple stack.
- (6) FAISS: library not DB, best for offline.
- (7) Chroma: embedded, dev-friendly.
- (8) Elasticsearch / OpenSearch: mature + hybrid search.
- Modern trend: pgvector for small teams, Qdrant/Milvus for scale.
How do you serve many fine-tuned LoRA adapters efficiently?
hard- Instead of separate models, share base weights + swap adapter per request.
- Techniques: (1) Multi-LoRA serving (vLLM, SGLang, LoRAX): dynamically apply adapter matrices per request.
- (2) Batched multi-adapter: different requests in same batch can use different adapters.
- (3) Adapter storage: MB per adapter vs GB per full fine-tune.
- Scale: serve 100+ specialized models on single GPU.
- Modern pattern for enterprise (per-customer fine-tune).
- Predibase LoRAX, vLLM LoRA support.
LLM router — what and why?
medium- Small model / rule-based classifier routes each request to appropriate LLM based on: (1) query complexity (simple → Haiku, complex → Opus).
- (2) domain (code → CodeLlama, chat → Claude).
- (3) latency budget.
- (4) cost constraint.
- (5) safety (harmful → refuse).
- Saves 30-70% cost with minimal quality loss when done well.
- Tools: RouteLLM, Martian, custom classifier + logistic regression.
- Popular in production.
- Trade-off: routing accuracy vs simplicity.
Your model must answer in 50 milliseconds. How do you allocate the budget?
medium- Measure the whole path first, because inference is rarely the dominant cost.
- Feature retrieval usually is, especially if it involves several network calls, so count them and fetch in parallel or precompute.
- Reserve headroom for the network and serialization, and budget against a high percentile rather than the mean, since the ninety-ninth percentile is what users experience and what times out.
- Then choose the model to fit what remains: a gradient boosted tree with a few hundred shallow trees answers in single-digit milliseconds, while a transformer generally does not without batching, quantization or a GPU.
- Cache aggressively where inputs repeat, and design a degraded path, such as a cached or default prediction, for when the budget is exceeded rather than letting the request hang.