Explain sinusoidal positional encoding.
mediumAnswer
- ; .
- Each dimension is a sinusoid of a different frequency.
- Advantage: encodes relative position (offset by k is a linear map on the encoding) and extrapolates to longer sequences than seen at training.
- Used in the original Transformer paper.
- Simpler learned-embedding variants replaced it in BERT / GPT-2 but modern LLMs favor RoPE / ALiBi.
Check yourself — multiple choice
- Random positional numbers
- Sin/cos of different frequencies per dim — encodes relative position, extrapolates to longer sequences
- Only one frequency used
- PE is learned in the original transformer
Sinusoidal PE: sin/cos at exponentially spaced frequencies → relative + extrapolation.
#positional-encoding#transformers
Practise Deep Learning
214 interview questions in this topic.