Network latency in ML serving — how much matters?

medium

Answer

$Total latency = client \to LB \to service \to feature fetch \to model \to response.$
$Model inference often only 30-50% of total.$
$Preprocess, feature store fetch, network hops dominate.$
$Optimize: (1) co-locate feature store with model (same AZ / region).$
$(2) precompute + cache features hot path.$
$(3) HTTP/2 or gRPC keep-alive (reduce connection setup).$
$\mathrm{off}\; + \;\mathrm{TCP}_{\mathrm{NODELAY}}$
$(5) Client-side connection pooling.$
$(6) Serverless has extra hop cost.$
$Profile every hop.$

Check yourself — multiple choice

Random
Total = client → LB → svc → feature fetch → model → resp; model often 30-50% only; optimize: co-locate FS + cache + HTTP/2 $\mathrm{keep} - \mathrm{alive}\; + \;\mathrm{TCP}_{\mathrm{NODELAY}}\; + \;\mathrm{client}$ pooling; profile every hop
Just model
Not real

Network latency: co-locate FS + cache + HTTP/2 + pool; profile hops.

#infrastructure#serving

215 interview questions in this topic.