EasyDeepLearn
LLMs & GenAI · section 1 of 18

Fundamentals

8 interview questions on fundamentals, each answered in full. Free to read, no account needed.

What is a large language model, in one paragraph?

easy
  • An LLM is a large transformer trained on massive text (and often code/multimodal data) to predict the next token given the previous ones.
  • Scale of parameters and data lets it acquire broad linguistic and world knowledge.
  • After pretraining, it is typically instruction-tuned and aligned (SFT + preference optimization such as RLHF or DPO) to behave as a helpful assistant.
  • At inference it generates tokens autoregressively from a prompt.
#fundamentalsPermalink & quiz →

Walk through how one training step of an autoregressive LLM works.

easy
  • Take a text batch, tokenize, forward the transformer to get logits for every position (predicting next token from each prefix).
  • Compute cross-entropy against the true next tokens with a causal mask so position t only sees positions  ≤t\mathrm{positions}\; \le t.
  • Sum losses over all positions (that's L-1 predictions per L-token sequence — huge signal per step, much cheaper than one prediction per sequence).
  • Backprop, AdamW step.
  • Repeat on trillions of tokens.
#fundamentals#pretrainingPermalink & quiz →

How does Byte-Pair Encoding (BPE) tokenization work?

medium
  • Start with a vocabulary of individual bytes/chars.
  • Repeatedly find the most frequent adjacent pair in the corpus and merge it into a new token, until the desired vocab size (e.g., 32k, 100k, 200k).
  • At inference, apply the same merge rules greedily.
  • Byte-level BPE (GPT-2+, Llama, Mistral) starts from raw bytes → handles any Unicode without OOV.
  • Common vocab sizes: 32k (Llama-2), 100k (GPT-4), 200k+ (Gemini).
  • Larger vocab = fewer tokens per text = more efficient long-context inference.
#tokenization#fundamentalsPermalink & quiz →

Why don't LLMs simply use character-level tokenization?

medium
  • Character-level: very long sequences (5-10x longer than BPE) → attention O(n2)O(n^{2}) becomes prohibitive.
  • Also, characters carry little semantic information — each attention step handles less content, requiring deeper / wider models.
  • BPE / SentencePiece with 32k-200k vocab balances sequence length vs semantic density.
  • Byte-level BPE gives the character-level robustness (any Unicode) without the sequence-length penalty.
#tokenization#fundamentalsPermalink & quiz →

What does 'emergence' mean in LLM capabilities and is it real?

hard
  • Original claim (Wei et al., 2022): some capabilities are absent at small scale and appear sharply above a compute threshold — 'phase transitions'.
  • Subsequent work (Schaeffer et al., 2023) showed many 'emergent' curves are artifacts of using discrete metrics (accuracy) instead of continuous ones (log-loss on the right sub-task); the underlying capability improves smoothly with scale.
  • Real capabilities like in-context learning, chain-of-thought, and code do improve smoothly with scale — 'emergence' is more measurement than magic.
#scaling#fundamentalsPermalink & quiz →

Why do modern LLMs use decoder-only architectures?

medium
  • (1) Simpler training objective (next-token) unifies pretraining and generation.
  • (2) Scales more predictably — no separate encoder to balance.
  • (3) Zero-shot / few-shot works out of the box via in-context learning.
  • (4) Encoder-decoder architectures (T5) don't scale as gracefully for open-ended generation.
  • (5) KV cache is straightforward with causal attention.
  • Encoder-only (BERT) is for classification / retrieval; decoder-only (GPT, Llama, Mistral) dominates general-purpose LLMs.
#architecture#fundamentalsPermalink & quiz →

What is SwiGLU and why do LLMs use it in the MLP block?

hard
  • SwiGLU (Shazeer 2020, adopted by PaLM, LLaMA, Mistral) replaces the standard MLP FF⁡(x)  =  W2  ⋅  GELU⁡(W1  ⋅  x)\operatorname{FF}(x)\; = \;W_{2}\; \cdot \;\operatorname{GELU}(W_{1}\; \cdot \;x) with FF⁡(x)  =  W2  ⋅  (Swish⁡(Wa  ⋅  x)  ⊙  (Wb  ⋅  x))\operatorname{FF}(x)\; = \;W_{2}\; \cdot \;(\operatorname{Swish}(W_{a}\; \cdot \;x)\; \odot \;(W_{b}\; \cdot \;x)).
  • The gated activation adds a multiplicative branch that lets the network selectively pass information.
  • Modest params overhead (~1.5x MLP), small but consistent quality gains (~1-2% perplexity).
  • Standard in all 2023+ open LLMs.
#architecture#fundamentalsPermalink & quiz →

What is in-context learning (ICL) and how does it work?

medium
  • Ability of LLMs to solve new tasks from examples provided in the prompt (few-shot) without weight updates.
  • E.g., show 3 (English, French) pairs and ask for a fourth translation.
  • The model performs 'meta-learning' during pretraining: it saw enough patterns to recognize a new task's structure from a handful of examples.
  • Mechanistic interpretation: attention heads implement an approximate gradient descent step over the in-context examples ('learning without weight updates' — Akyürek et al. 2022).
#prompting#in-context-learning#fundamentalsPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz