EasyDeepLearn
LLMs & GenAI · section 4 of 18

Architecture & attention variants

8 interview questions on architecture & attention variants, each answered in full. Free to read, no account needed.

What is the context window and what tricks extend it?

medium
  • The context window is the maximum number of tokens (prompt + response) the model can attend to.
  • Extending it: RoPE scaling (position interpolation, YaRN), sparse/linear attention, sliding-window attention, retrieval-augmented long-context, and KV-cache tricks.
  • Longer context helps but is not free — attention is quadratic in sequence length, and quality often degrades far from the beginning ('lost in the middle').
#long-context#architecture#inferencePermalink & quiz →

What is a Mixture-of-Experts (MoE) LLM?

hard
  • An MoE model has many expert subnetworks (usually MLP blocks) and a router that picks a few (top-k, often 2) per token.
  • Only the selected experts run — the model has huge total parameter count but far fewer active parameters per token, so compute stays close to a dense model of the smaller active size.
  • Used by Mixtral, DeepSeek-V3, and others.
  • Challenges: load balancing across experts, communication overhead in distributed training.
#moe#architecturePermalink & quiz →

Why did RoPE replace learned positional embeddings in modern LLMs?

hard
  • (1) RoPE encodes relative position — the attention score between positions i and j depends only on i-j, which is a better match for language structure.
  • (2) It extrapolates gracefully to longer sequences than seen at training (with NTK-aware / YaRN scaling).
  • (3) No positional-embedding parameters to memorize a fixed range.
  • (4) Easily integrates with Flash Attention.
  • All modern open LLMs use RoPE (LLaMA, Mistral, Falcon, Qwen); GPT-4 is speculated to as well.
#architecturePermalink & quiz →

What is Grouped-Query Attention (GQA) and why does it matter?

hard
  • Standard multi-head attention: each head has its own K and V — the KV cache is heads × dheadd_{\mathrm{head}} × seq × batch.
  • MQA (Shazeer 2019): all heads share one K/V — much smaller KV cache, ~1% quality loss.
  • GQA (Ainslie 2023): heads are grouped (e.g., 8 groups of 4 heads), each group shares K/V — 4-8x smaller KV cache, negligible quality loss.
  • Used by LLaMA-2 70B, LLaMA-3, Mistral, Qwen.
  • Critical for long-context inference where KV cache dominates VRAM.
#architecture#attention#kv-cache#inferencePermalink & quiz →

How does Mixtral 8x7B differ from a dense LLM?

hard
  • Mixtral has 8 expert MLP blocks per layer and a top-2 router.
  • Total params: ~46B (8 experts × ~5.5B expert MLP + shared attention).
  • Active per token: ~13B (2 experts + attention).
  • Trained with load-balancing auxiliary loss and expert-parallel data-parallelism.
  • Result: quality comparable to LLaMA-2 70B at ~4x cheaper inference (only active params matter for forward-pass FLOPs).
  • Trade-off: total VRAM footprint is still ~46B.
#moe#architecturePermalink & quiz →

How does YaRN extend an LLM's context beyond training length?

hard
  • YaRN (Peng et al., 2023) is a RoPE frequency-rescaling technique.
  • Split the RoPE frequency dimensions into three regions by their wavelength: high-frequency dims are left untouched (short-range), low-frequency dims are linearly interpolated (long-range), mid-range gets a smooth interpolation.
  • Add a per-frequency temperature adjustment on attention logits to counteract the entropy shift from position scaling.
  • Fine-tune for ~1B tokens with the new scale.
  • Result: LLaMA-2 → 128k context at minimal quality loss.
#long-context#architecturePermalink & quiz →

How does Flash Attention 2 speed up training?

hard
  • Flash Attention (Dao 2022) tiles Q, K, V into blocks fitting in SRAM and uses online softmax to compute exact attention with O(n) memory instead of O(n2)O(n^{2}) — 2-4x speedup.
  • Flash Attention 2 (Dao 2023) restructures the parallelism: parallelizes along sequence length in the forward pass and along heads/batch in the backward pass, uses fewer non-matmul FLOPs, achieves 50-70% of theoretical GPU throughput.
  • Standard kernel in modern LLM training and inference.
#architecture#attention#inferencePermalink & quiz →

How is a modern vision-language model (LLaVA, GPT-4V, Claude 3) built?

hard
  • (1) Pretrained vision encoder (CLIP-ViT or SigLIP) produces image feature tokens.
  • (2) Projector (MLP or Q-former) maps vision tokens into the LLM's embedding space.
  • (3) Feed vision tokens as a prefix to a pretrained LLM decoder.
  • (4) Train: (a) freeze both encoders, train only the projector on image-caption pairs; (b) then fine-tune LLM + projector on visual instruction data (LLaVA-Instruct).
  • Modern additions: interleaved image-text pretraining, dynamic image resolution, video frames.
  • GPT-4V, Claude 3, Gemini follow this pattern at scale.
#multimodal#architecturePermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz