Why does a transformer need positional encoding?
hardAnswer
- Self-attention is permutation-invariant — without position info, a sentence and its shuffle look identical.
- Positional encoding injects order.
- Options: fixed sinusoidal (original Transformer), learned absolute embeddings (BERT, GPT-2), relative positional bias (T5), and rotary position embeddings — RoPE — used in LLaMA, Mistral, and most modern LLMs.
Check yourself — multiple choice
- Attention is naturally position-aware
- Positional encodings are needed because self-attention is permutation-invariant
- Modern LLMs avoid all positional encodings
- Sinusoidal encodings are learned during training
Without positions, attention treats a sequence as a set.
#positional-encoding#transformers
Practise Deep Learning
214 interview questions in this topic.