EasyDeepLearn

How does Rotary Position Embedding (RoPE) work?

hard

Answer

  • Rotate the Q and K vectors in each 2D subspace of the head dimension by an angle proportional to the position: R(θpos)  ⋅  qR({\theta}_{\mathrm{pos}})\; \cdot \;q, R(θpos)  ⋅  kR({\theta}_{\mathrm{pos}})\; \cdot \;k, with θpos  =  pos  /  base{\theta}_{\mathrm{pos}}\; = \;\mathrm{pos}\; / \;\mathrm{base}^(2i/d).
  • Because of the rotation identity, (R(θi)q)(R({\theta}_{i})q) · (R(θj)k)(R({\theta}_{j})k) depends only on the difference θi  −  θj{\theta}_{i}\; - \;{\theta}_{j} — encodes relative position multiplicatively.
  • Extrapolates well (with NTK / linear scaling tweaks), used in LLaMA, Mistral, Falcon, Qwen, GPT-NeoX, most modern open LLMs.
Check yourself — multiple choice
  • Add to embeddings
  • Rotate Q, K per position by θpos  =  pos/base{\theta}_{\mathrm{pos}}\; = \;\mathrm{pos} / \mathrm{base}^(2i/d) — inner product depends only on (i - j); standard in modern LLMs
  • RoPE is deprecated
  • Same as ALiBi

RoPE: 2D rotations of Q, K → dot product depends on relative position.

#positional-encoding#transformers

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions