EasyDeepLearn

What is SwiGLU and why do LLMs use it in the MLP block?

hard

Answer

  • SwiGLU (Shazeer 2020, adopted by PaLM, LLaMA, Mistral) replaces the standard MLP FF⁡(x)  =  W2  ⋅  GELU⁡(W1  ⋅  x)\operatorname{FF}(x)\; = \;W_{2}\; \cdot \;\operatorname{GELU}(W_{1}\; \cdot \;x) with FF⁡(x)  =  W2  ⋅  (Swish⁡(Wa  ⋅  x)  ⊙  (Wb  ⋅  x))\operatorname{FF}(x)\; = \;W_{2}\; \cdot \;(\operatorname{Swish}(W_{a}\; \cdot \;x)\; \odot \;(W_{b}\; \cdot \;x)).
  • The gated activation adds a multiplicative branch that lets the network selectively pass information.
  • Modest params overhead (~1.5x MLP), small but consistent quality gains (~1-2% perplexity).
  • Standard in all 2023+ open LLMs.
Check yourself — multiple choice
  • Same as GELU MLP
  • Gated activation FF⁡(x)  =  W2\operatorname{FF}(x)\; = \;W_{2}·(SiLU⁡(Wa⋅x)  ⊙  (Wb⋅x))(\operatorname{SiLU}(W_{a} \cdot x)\; \odot \;(W_{b} \cdot x)) — small quality gain, standard in modern LLMs
  • SwiGLU removes MLP
  • Only for CNNs

SwiGLU: gated SiLU MLP → small consistent quality gain, standard since PaLM.

#architecture#fundamentals

Practise LLMs & GenAI

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions