Serverless ML inference — pros / cons?
mediumAnswer
- AWS Lambda / Cloud Run / Cloud Functions run model on demand.
- Pros: (1) pay per invocation (great for spiky / low traffic).
- (2) auto-scale to zero.
- (3) no infra to manage.
- Cons: (1) cold start (100ms-30s depending on model size + runtime).
- (2) memory / duration limits (Lambda 10 GB / 15 min).
- (3) no GPU on most (Lambda has limited GPU).
- (4) more expensive per prediction at high traffic.
- Rule: <10 QPS + latency-tolerant → serverless; else provisioned.
Check yourself — multiple choice
- Random
- Serverless (Lambda/Cloud Run/Functions): pay per invoke + auto-scale-to-zero + no infra; downside cold start + memory/time limits + limited GPU + expensive at scale; <10 QPS latency-tolerant → serverless
- Always best
- Not real
Serverless: spiky low-traffic + latency-tolerant; not for high QPS or GPU.
#serving#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.