How does relative positional encoding (Shaw / T5) work?

hard

Answer

$Instead of adding a positional vector to the token embedding, inject position information into the attention score itself as a learned bias depending on the offset (i - j) between query and key.$
$In T5, this bias is bucketed (log-spaced) so the number of learned parameters stays finite.$
$Handles arbitrarily long sequences reasonably well, but requires custom attention kernels — less common than RoPE in modern LLMs.$

Check yourself — multiple choice

Add position to embeddings
Learned bias on attention score based on the (i - j) offset — handles long sequences via bucketing (T5)
Only sinusoidal
Same as RoPE

Relative PE: attention-score bias depending on offset (T5 uses log-bucketed offsets).

#positional-encoding#transformers

214 interview questions in this topic.