What is embedding tying and when do LLMs use it?
mediumAnswer
- Share the same weight matrix between the input token embedding and the output LM head.
- Saves ~V*d parameters (128k * 4k = 500M for a small LLM — huge share).
- Slightly hurts quality on very large models (untied heads specialize output distribution).
- Used in smaller LLMs (LLaMA-2 7B ties them; LLaMA-2 70B does not).
- For very-large-vocab models it's often kept untied to preserve modeling flexibility.
Check yourself — multiple choice
- Always tied
- Share weights between input embedding and LM head — saves V·d params, small quality cost; small LLMs tie, large ones often don't
- Impossible to share
- Only for CNNs
Embedding tying: share input/output projection weights → save params, slight quality cost.
#architecture#pretraining
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you prevent MoE routing collapse?
- Why is the softmax + cross-entropy at the LM head a training bottleneck?
- What parallelism strategies are combined to train a 70B model?
- What is the context window and what tricks extend it?
- What is a Mixture-of-Experts (MoE) LLM?
- Walk through how one training step of an autoregressive LLM works.