What SLIs / SLOs are typical for LLM serving?
mediumAnswer
- SLIs: TTFT P50/P95 (time-to-first-token), P95 tokens/second during decode, request success rate, tool-call success rate.
- SLOs (typical for chat): TTFT P95 < 1s, decode > 30 tok/s, availability 99.9%.
- Track cost/request too.
- Monitoring stack: Datadog / Grafana for latency, Prometheus for GPU metrics, W&B / Langfuse for LLM-specific traces (prompts, retrievals, tool calls, hallucination flags).
- Trace per-request end-to-end for debugging.
Check yourself — multiple choice
- No SLOs
- TTFT P95, decode tok/s, success rate, cost/req; targets like TTFT P95 <1s, decode >30 tok/s, 99.9% availability; per-request traces
- Same as REST APIs
- Impossible to measure
LLM SLOs: TTFT / decode speed / success rate + per-request tracing (Langfuse / W&B).
#serving#production#latency
Practise LLMs & GenAI
214 interview questions in this topic.