How much can you interpret a model from attention weights?
hardAnswer
- Attention weights are a noisy signal for interpretability.
- High weight ≠ 'the model uses this token'.
- Multiple heads look at overlapping things; residual streams carry lots of info regardless of attention.
- Better tools: attribution methods (integrated gradients, attention rollout), probing linear classifiers on hidden states, mechanistic interpretability (Anthropic's circuits).
- Attention maps are a starting point, not a proof.
Check yourself — multiple choice
- Attention weights are perfect explanations
- Attention is a noisy interpretability signal — use gradients, probes, mechanistic tools too
- Attention weights are random
- Attention interpretability is a solved problem
Attention weights ≠ explanation — noisy signal; complement with gradients/probes/circuits.
#transformers#attention#interpretability
Practise Deep Learning
214 interview questions in this topic.