22 interview questions on attention & transformers, each answered in full. Free to read, no account needed.
How does self-attention work in a transformer?
hard- For each token, produce a Query, Key, and Value vector.
- The output for token i is a weighted sum of all Values, where the weight is softmax(Qi⋅Kj/sqrt(dk)).
- Multi-head attention runs this in parallel with different projections and concatenates the results.
- Attention lets every token look at every other token — capturing long-range dependencies with O(n2) complexity.
Why does a transformer need positional encoding?
hard- Self-attention is permutation-invariant — without position info, a sentence and its shuffle look identical.
- Positional encoding injects order.
- Options: fixed sinusoidal (original Transformer), learned absolute embeddings (BERT, GPT-2), relative positional bias (T5), and rotary position embeddings — RoPE — used in LLaMA, Mistral, and most modern LLMs.
Why is attention scaled by 1/sqrt(dk)?
hard- Assume q, k have entries with mean 0 and variance 1 (post-init or post-normalization).
- Their dot product has variance dk.
- As dk grows (say, 64+), un-scaled dot products become large in magnitude, pushing softmax into the saturated regime where one weight is ~1 and the rest ~0 → tiny gradients w.r.t. the losing keys.
- Dividing by sqrt(dk) rescales the variance back to 1 and keeps softmax gradients healthy.
Why multi-head attention instead of a single big head?
medium- Different heads can attend to different types of relationships (syntactic, semantic, positional) in different subspaces.
- Multi-head with H heads of dim d/H has similar compute to one head of dim d, but the diversity of learned attention patterns empirically helps a lot.
- Also enables interpretation via per-head attention maps.
- Typical: 8-32 heads for d ~ 512-4096.
How do you pick the head dimension dhead?
hard- Common choice: dhead=d/nheads (so total FLOPs match a single-head version with hidden dim d). dhead too small (< 32) → heads can't represent complex relationships; too large (> 128) → wastes parameters on redundancy.
- Typical: dhead=64−128.
- Grouped-query attention (GQA, MQA) share K/V across groups of heads to reduce KV cache without hurting quality much.
Why is standard self-attention O(n2) in sequence length?
medium- For each of n query tokens, compute a dot product with each of n key tokens → n2 dot products, each of size d.
- Then a softmax over n weights per query.
- Both compute and memory are O(n2⋅d).
- Fine at n = 512-2048; painful at n = 32k+.
- Motivates efficient / sparse / linear-attention variants and paging tricks (Flash Attention doesn't reduce complexity but cuts memory dramatically via tiling).
How do Performer / Linformer / linear attention reduce O(n2)?
hard- Performer: replace exp(q⋅k/sqrt(d)) with a positive-feature kernel φ(q)·φ(k) that lets you re-associate the matmul as (KTV)(QT) — O(n⋅d2) instead of O(n2⋅d).
- Linformer: project the sequence-length dimension of K, V from n to a small constant k → O(n * k * d).
- Trade-offs: approximation error, sometimes worse quality on long-range benchmarks vs full attention.
What is sparse / sliding-window attention?
medium- Restrict each token to attend to a fixed neighborhood (window of size w) rather than all n tokens.
- Complexity drops from O(n2) to O(n * w).
- Stack layers so the effective receptive field grows with depth.
- Used in Longformer (window + a few global tokens), BigBird, Mistral's sliding window attention.
- Great when relevant context is local.
- Combine with a few globally-attending tokens for long-range info.
How do Longformer / BigBird combine sparse and global attention?
hard- Combine local sliding-window attention (each token sees a window of size w) with a handful of 'global' tokens (e.g., [CLS], question tokens in QA) that attend to every position and are attended by every position.
- BigBird adds a random-attention pattern for provably close approximation of full attention.
- Enables 4k-16k context on hardware where full attention would OOM.
What is Flash Attention?
hard- Dao et al. (2022) IO-aware attention algorithm: tile Q, K, V into blocks that fit in on-chip SRAM, compute softmax incrementally with the online softmax trick, and never materialize the full n×n attention matrix in HBM.
- Same math as standard attention (exact, not approximate), but 2-4x faster and O(n) memory instead of O(n2).
- Enabled longer contexts (32k, 100k) on the same hardware.
- Flash-Attention-2 improves parallelism further.
What is the KV cache in transformer inference?
hard- During autoregressive generation, tokens are produced one at a time.
- For each layer, the K and V projections of all previously generated tokens are cached; only the new token's Q, K, V are computed each step.
- Time per new token drops from O(n2) to O(n).
- Memory cost: 2⋅nlayers⋅nheads⋅dhead⋅seqlen per sequence — often dominates GPU memory in long-context serving.
- Techniques: MQA/GQA (share K/V across heads), Paged attention (vLLM), sliding-window caches.
Explain sinusoidal positional encoding.
medium- PE(pos,2i)=sin(pos/10000(2i/d)); PE(pos,2i+1)=cos(pos/10000(2i/d)).
- Each dimension is a sinusoid of a different frequency.
- Advantage: encodes relative position (offset by k is a linear map on the encoding) and extrapolates to longer sequences than seen at training.
- Used in the original Transformer paper.
- Simpler learned-embedding variants replaced it in BERT / GPT-2 but modern LLMs favor RoPE / ALiBi.
What is the main limitation of learned absolute positional embeddings?
medium- They can't extrapolate — at inference, positions beyond maxpositionembeddings have never been trained and produce garbage.
- Also, they encode absolute rather than relative positions, so translating the sentence within a longer context changes the embedding.
- Solved by relative PE (Shaw), RoPE (rotational), or ALiBi (linear bias on attention scores) — all of which behave well at longer sequences.
How does relative positional encoding (Shaw / T5) work?
hard- Instead of adding a positional vector to the token embedding, inject position information into the attention score itself as a learned bias depending on the offset (i - j) between query and key.
- In T5, this bias is bucketed (log-spaced) so the number of learned parameters stays finite.
- Handles arbitrarily long sequences reasonably well, but requires custom attention kernels — less common than RoPE in modern LLMs.
How does ALiBi encode position?
hard- Attention with Linear Biases (Press et al., 2021): add a linear penalty proportional to distance to the attention logits: ai,j -= mh⋅(i−j), where mh is a fixed per-head slope.
- No learned embeddings — just a static penalty.
- Extrapolates naturally to longer sequences at inference than at training.
- Simple, effective, used in BLOOM and some other LLMs; RoPE dominates elsewhere.
How does Rotary Position Embedding (RoPE) work?
hard- Rotate the Q and K vectors in each 2D subspace of the head dimension by an angle proportional to the position: R(θpos)⋅q, R(θpos)⋅k, with θpos=pos/base^(2i/d).
- Because of the rotation identity, (R(θi)q) · (R(θj)k) depends only on the difference θi−θj — encodes relative position multiplicatively.
- Extrapolates well (with NTK / linear scaling tweaks), used in LLaMA, Mistral, Falcon, Qwen, GPT-NeoX, most modern open LLMs.
How does cross-attention differ from self-attention?
medium- Cross-attention: queries come from one sequence (e.g., decoder tokens), keys/values from another (e.g., encoder outputs, retrieved documents, image patches).
- The decoder 'reads' the encoder representation without needing its own tokens to encode it.
- Used in NMT decoder (attend to encoder), CLIP-style multimodal retrieval, retrieval-augmented generation, image captioning.
- Self-attention has Q, K, V all from the same sequence.
How much can you interpret a model from attention weights?
hard- Attention weights are a noisy signal for interpretability.
- High weight ≠ 'the model uses this token'.
- Multiple heads look at overlapping things; residual streams carry lots of info regardless of attention.
- Better tools: attribution methods (integrated gradients, attention rollout), probing linear classifiers on hidden states, mechanistic interpretability (Anthropic's circuits).
- Attention maps are a starting point, not a proof.
Briefly, what do BLEU and ROUGE measure?
medium- BLEU: n-gram precision of the generated hypothesis against one or more reference translations, with a brevity penalty for short outputs.
- Standard for machine translation.
- ROUGE: n-gram recall of the summary against reference summaries (ROUGE-N for n-grams, ROUGE-L for longest common subsequence).
- Standard for summarization.
- Both are crude — modern metrics use learned embeddings (BERTScore, BLEURT, COMET) or LLM-as-judge.
What are NTK-aware RoPE scaling and YaRN?
hard- Techniques to extend a RoPE-based LLM's context window beyond the training length without retraining from scratch.
- Naive linear interpolation of positions (Position Interpolation) shrinks the effective frequency and hurts short-range attention.
- NTK-aware scaling adjusts higher frequencies less than lower ones (motivated by the Neural Tangent Kernel view).
- YaRN (Peng et al., 2023) refines this with per-frequency temperature and a small fine-tune — lets LLaMA-2 extend from 4k to 128k tokens at minimal quality loss.
How do you extend a transformer to very long contexts?
hard- Options: (1) sparse/local attention (Longformer, Mistral's sliding window); (2) linear-attention approximations (Performer, RWKV, Mamba SSM); (3) retrieval augmentation (chunk documents, retrieve top-k with a vector store, stuff into the prompt); (4) position-encoding extrapolation (RoPE scaling, NTK-aware interpolation, YaRN); (5) hierarchical models (encode chunks separately then attend at a higher level).
- Combine multiple techniques in practice.
Attention is quadratic in sequence length. Why is FlashAttention still a major win without changing that?
hard- Because the practical bottleneck is memory traffic, not arithmetic.
- A naive implementation materializes the full attention matrix in high-bandwidth memory, so the cost is dominated by writing and reading a quadratic-sized intermediate.
- FlashAttention tiles the computation, keeps blocks in on-chip memory, and computes the softmax in a streaming, numerically stable way, so the quadratic matrix never exists in memory at all.
- Complexity stays quadratic in time but memory becomes linear in sequence length, and wall-clock speed improves several-fold because the kernel is no longer memory-bound.
- It is exact, unlike sparse or low-rank approximations, which is why it was adopted universally rather than as a quality tradeoff.