What is a 'model router' and when is it worth it?
hardAnswer
- Cheap classifier that decides which downstream LLM to send each request to based on difficulty / cost / latency.
- E.g., simple factual questions → 8B cheap model; complex reasoning → 70B or GPT-4; multimodal → a vision-language model.
- Router can be a small transformer, a linear probe on embeddings, or an LLM-as-router.
- Rationale: 60-80% of production traffic doesn't need frontier quality — routing saves 3-10× on average cost.
- Trade-off: routing errors send hard queries to weak models.
- Route on a small % and A/B.
Check yourself — multiple choice
- Always use the biggest model
- Route by difficulty/cost: cheap queries to small models, hard ones to frontier — 3-10× average cost cut with A/B safeguards
- Only for training
- Impossible
Model router: cheap classifier picks the right-sized LLM → big cost win when tuned + A/B'd.
#production#cost#serving
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you pick a GPU for serving a 70B LLM?
- Why is cold-starting an LLM slow and how do you fix it?
- How much does it cost to pretrain a modern LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?