How does cross-attention differ from self-attention?
mediumAnswer
- Cross-attention: queries come from one sequence (e.g., decoder tokens), keys/values from another (e.g., encoder outputs, retrieved documents, image patches).
- The decoder 'reads' the encoder representation without needing its own tokens to encode it.
- Used in NMT decoder (attend to encoder), CLIP-style multimodal retrieval, retrieval-augmented generation, image captioning.
- Self-attention has Q, K, V all from the same sequence.
Check yourself — multiple choice
- Cross-attention has no Q, K, V
- Q from one sequence, K/V from another — decoder reads encoder in NMT, retrieval, multimodal
- Same as self-attention
- Only used in CNNs
Cross-attention: Q from A, K/V from B → decoder reads encoder (or retrieved docs).
#attention#transformers
Practise Deep Learning
214 interview questions in this topic.