How does ALiBi encode position?
hardAnswer
- Attention with Linear Biases (Press et al., 2021): add a linear penalty proportional to distance to the attention logits: -= , where is a fixed per-head slope.
- No learned embeddings — just a static penalty.
- Extrapolates naturally to longer sequences at inference than at training.
- Simple, effective, used in BLOOM and some other LLMs; RoPE dominates elsewhere.
Check yourself — multiple choice
- Learned embeddings
- Static linear bias on attention scores by distance — no params, extrapolates naturally
- Random
- Same as sinusoidal
ALiBi: linear distance penalty in attention → no params + extrapolation.
#positional-encoding#transformers
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why does a transformer need positional encoding?
- Explain sinusoidal positional encoding.
- What is the main limitation of learned absolute positional embeddings?
- How does relative positional encoding (Shaw / T5) work?
- How does Rotary Position Embedding (RoPE) work?
- What are NTK-aware RoPE scaling and YaRN?