When do you distill an LLM for serving?
mediumAnswer
- When latency / cost constraints require a smaller model than the frontier.
- Train a smaller student on outputs from a larger teacher, either via: (1) SFT on teacher's greedy outputs (imitation); (2) matching teacher's logit distribution at high temperature (KD loss); (3) mining teacher's chain-of-thought traces.
- Typical setup: distill a 70B teacher into a 7B / 13B student for 90-95% of the quality at 10× cheaper serving.
- DistilBERT (2019) was the first big example; today used everywhere (Zephyr, Qwen, Phi).
Check yourself — multiple choice
- Never distill for serving
- Small student trained on large teacher's outputs / logits / CoT → 90-95% quality at 10× cheaper inference
- Only for training
- Same as fine-tuning
Distill 70B teacher → 7B student for cheaper serving at ~90-95% quality.
#compression#inference#cost
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is quantization and what tradeoffs come with it?
- What is prompt caching and when does it help?
- What does 'automatic prefix caching' do in vLLM?
- How do you pick a GPU for serving a 70B LLM?
- Your LLM feature costs too much per request. What levers do you pull, in order?
- What is the context window and what tricks extend it?