What is a Mixture-of-Experts (MoE) LLM?
hardAnswer
- An MoE model has many expert subnetworks (usually MLP blocks) and a router that picks a few (top-k, often 2) per token.
- Only the selected experts run — the model has huge total parameter count but far fewer active parameters per token, so compute stays close to a dense model of the smaller active size.
- Used by Mixtral, DeepSeek-V3, and others.
- Challenges: load balancing across experts, communication overhead in distributed training.
Check yourself — multiple choice
- MoE activates all experts for every token
- MoE routes each token to a small subset of experts
- MoE always improves latency over dense models
- MoE has no load-balancing issues
Only a top-k subset of experts activates per token — sparse computation.
#moe#architecture
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How does Mixtral 8x7B differ from a dense LLM?
- How do you prevent MoE routing collapse?
- What is the context window and what tricks extend it?
- Why do modern LLMs use decoder-only architectures?
- What is SwiGLU and why do LLMs use it in the MLP block?
- Why did RoPE replace learned positional embeddings in modern LLMs?