How do you prevent MoE routing collapse?
hardAnswer
- Without regularization, the router tends to funnel most tokens to a small subset of experts (routing collapse), starving the others.
- Fixes: (1) auxiliary load-balancing loss penalizing uneven per-expert token counts within a batch; (2) capacity limit — each expert accepts at most 1.25x its 'fair share' of tokens per batch, excess is dropped or routed to other experts; (3) noisy top-k gating (add Gumbel noise) for exploration; (4) expert dropout during training.
Check yourself — multiple choice
- Nothing needed
- Aux load-balance loss + capacity cap + noisy gating + expert dropout to prevent routing collapse
- Increase learning rate
- MoE naturally balances
MoE balancing: load-balance aux loss + capacity + noisy gating.
#moe#architecture#pretraining
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is a Mixture-of-Experts (MoE) LLM?
- How does Mixtral 8x7B differ from a dense LLM?
- Why is the softmax + cross-entropy at the LM head a training bottleneck?
- What parallelism strategies are combined to train a 70B model?
- What is embedding tying and when do LLMs use it?
- What is the context window and what tricks extend it?