Why does a transformer need positional encoding?
hardAnswer
- Self-attention is permutation-invariant — without position info, a sentence and its shuffle look identical.
- Positional encoding injects order.
- Options: fixed sinusoidal (original Transformer), learned absolute embeddings (BERT, GPT-2), relative positional bias (T5), and rotary position embeddings — RoPE — used in LLaMA, Mistral, and most modern LLMs.
How to say it out loud
Self-attention is permutation invariant. If you shuffle the tokens, every attention score comes out the same, so the model literally cannot separate 'dog bites man' from 'man bites dog'. Positional encoding is how you put the order back in. The original paper used fixed sinusoids, BERT and GPT-2 learned absolute embeddings, T5 adds a relative bias to the attention logits, and essentially every current LLM uses RoPE. The reason RoPE won is that it rotates queries and keys by an angle proportional to position, so the dot product ends up depending on the distance between two tokens rather than their absolute indices — which is what lets it extrapolate past the training context.
Check yourself — multiple choice
- Attention is naturally position-aware
- Positional encodings are needed because self-attention is permutation-invariant
- Modern LLMs avoid all positional encodings
- Sinusoidal encodings are learned during training
Without positions, attention treats a sequence as a set.
What the interviewer asks next
- How does self-attention work in a transformer?The permutation-invariance claim is only convincing if you can write the equation it follows from.
- Why is attention scaled by 1/sqrt(d_k)?The other 'why is this term here' question about the same formula, and it comes up just as often.
- What are NTK-aware RoPE scaling and YaRN?Once you say RoPE, the follow-up is how you stretch it beyond the length it was trained on.
- How does a Vision Transformer (ViT) treat images?ViT needs position information too, and for a reason worth contrasting with text.
Practise Deep Learning
214 interview questions in this topic.