LLM router — what and why?
mediumAnswer
- Small model / rule-based classifier routes each request to appropriate LLM based on: (1) query complexity (simple → Haiku, complex → Opus).
- (2) domain (code → CodeLlama, chat → Claude).
- (3) latency budget.
- (4) cost constraint.
- (5) safety (harmful → refuse).
- Saves 30-70% cost with minimal quality loss when done well.
- Tools: RouteLLM, Martian, custom classifier + logistic regression.
- Popular in production.
- Trade-off: routing accuracy vs simplicity.
Check yourself — multiple choice
- Random
- Small model/rule classifier routes to appropriate LLM by complexity/domain/latency/cost/safety; saves 30-70% cost with minimal quality loss; RouteLLM / Martian / custom classifier
- One model
- Not real
Router: classify + dispatch; 30-70% cost save; RouteLLM / Martian.
#llmops#serving
Practise MLOps & Data Quality
215 interview questions in this topic.