How do you canary-deploy a new LLM version?
mediumAnswer
- (1) Route 1-5% of traffic to the new model, 95-99% to the current one.
- (2) A/B compare: quality metrics (LLM-judge, human eval on sampled outputs), latency, cost, error rate.
- (3) Include a fallback path if the canary throws or misbehaves.
- (4) Compare per-cohort (locale, tenant, use case) — regressions often show in subsets.
- (5) Roll forward gradually (5% → 25% → 50% → 100%) over hours / days with automated rollback triggers.
- Never a big-bang cutover on production LLM changes.
Check yourself — multiple choice
- Full switch immediately
- 1-5% canary + A/B compare quality/latency/cost + per-cohort + gradual rollout + auto-rollback
- Only offline eval
- No traffic split needed
LLM canary: small % traffic + A/B + gradual rollout + auto-rollback.
#production#serving
Practise LLMs & GenAI
214 interview questions in this topic.