How does Mixtral 8x7B differ from a dense LLM?
hardAnswer
- Mixtral has 8 expert MLP blocks per layer and a top-2 router.
- Total params: ~46B (8 experts × ~5.5B expert MLP + shared attention).
- Active per token: ~13B (2 experts + attention).
- Trained with load-balancing auxiliary loss and expert-parallel data-parallelism.
- Result: quality comparable to LLaMA-2 70B at ~4x cheaper inference (only active params matter for forward-pass FLOPs).
- Trade-off: total VRAM footprint is still ~46B.
Check yourself — multiple choice
- Same as dense LLaMA
- 8 MLP experts per layer, top-2 routing → 46B total / 13B active; matches LLaMA-70B quality at ~4× cheaper compute
- MoE removes attention
- Only one expert per layer
Mixtral: 8 experts, top-2 routing → dense-70B quality at ~13B active-params cost.
#moe#architecture
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is a Mixture-of-Experts (MoE) LLM?
- How do you prevent MoE routing collapse?
- What is the context window and what tricks extend it?
- Why do modern LLMs use decoder-only architectures?
- What is SwiGLU and why do LLMs use it in the MLP block?
- Why did RoPE replace learned positional embeddings in modern LLMs?