EasyDeepLearn

What is the main limitation of learned absolute positional embeddings?

medium

Answer

  • They can't extrapolate — at inference, positions beyond max⁡positionembeddings\operatorname{max}_{\mathrm{position}}\mathrm{embeddings} have never been trained and produce garbage.
  • Also, they encode absolute rather than relative positions, so translating the sentence within a longer context changes the embedding.
  • Solved by relative PE (Shaw), RoPE (rotational), or ALiBi (linear bias on attention scores) — all of which behave well at longer sequences.
Check yourself — multiple choice
  • Learned PE extrapolates perfectly
  • Can't extrapolate beyond max⁡position\operatorname{max}_{\mathrm{position}} and encodes absolute (not relative) position — RoPE/ALiBi solve both
  • Learned PE is the modern standard
  • Same as sinusoidal PE

Learned absolute PE fails past max⁡position\operatorname{max}_{\mathrm{position}} and encodes absolute — RoPE/ALiBi fix both.

#positional-encoding#transformers

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions