EasyDeepLearn
LLMs & GenAI · section 3 of 18

Pretraining & scaling

16 interview questions on pretraining & scaling, each answered in full. Free to read, no account needed.

What data goes into modern LLM pretraining?

medium
  • A mix of: (1) filtered CommonCrawl web text (60-80%), (2) high-quality curated corpora — books, Wikipedia, arXiv, StackExchange (5-20%), (3) code — GitHub, competitive programming, docs (5-20%), (4) mathematical text and problem sets.
  • Multi-lingual coverage.
  • Aggressive deduplication (near-duplicate removal at document, paragraph, and n-gram levels), quality filtering (classifier trained on Wikipedia vs random web), and PII / toxic content removal.
  • Data quality > data quantity beyond a point.
#pretraining#dataPermalink & quiz →

Why is data deduplication critical for LLM pretraining?

medium
  • Duplicate documents cause the model to memorize verbatim rather than generalize (Carlini et al., 2020).
  • Effects: worse generalization, higher memorization risk (training data extraction attacks), inflated benchmark scores if test data leaks via duplicates, wasted compute.
  • Modern pretraining pipelines run exact-hash dedup + fuzzy dedup (MinHash / SimHash on shingles) at document and paragraph levels — often removing 20-40% of the raw corpus.
#pretraining#dataPermalink & quiz →

Summarize Kaplan et al. (2020) scaling laws in one sentence.

hard
  • Loss scales as a power law in three variables — parameters N, dataset size D, and compute C — with each factor showing predictable diminishing returns.
  • Kaplan concluded compute-optimal training used relatively large N and small D; Chinchilla (Hoffmann 2022) corrected this by fitting jointly and showed the optimum uses roughly 20 tokens per parameter — bigger data than Kaplan predicted.
#pretraining#scalingPermalink & quiz →

State the Chinchilla scaling insight and its practical impact.

hard
  • For compute C ~= 6*N*D, the compute-optimal split gives N and D roughly proportional (~20 tokens per parameter).
  • Older LLMs like GPT-3 (175B, 300B tokens = 1.7 tokens/param) were massively undertrained.
  • Practical shift: modern open LLMs (LLaMA-2 7B on 2T tokens = 285 tokens/param; LLaMA-3 8B on 15T = 1875 tokens/param) train small models on huge data — cheaper inference, stronger quality per active parameter.
#pretraining#scalingPermalink & quiz →

How do you prevent MoE routing collapse?

hard
  • Without regularization, the router tends to funnel most tokens to a small subset of experts (routing collapse), starving the others.
  • Fixes: (1) auxiliary load-balancing loss penalizing uneven per-expert token counts within a batch; (2) capacity limit — each expert accepts at most 1.25x its 'fair share' of tokens per batch, excess is dropped or routed to other experts; (3) noisy top-k gating (add Gumbel noise) for exploration; (4) expert dropout during training.
#moe#architecture#pretrainingPermalink & quiz →

How do you train an LLM to a long context length?

hard
  • Pretrain on shorter sequences (4-8k) for most of training — cheaper and more diverse.
  • Then extend the context in a final continued-pretraining phase: increase max⁡seqlen\operatorname{max}_{\mathrm{seq}}\mathrm{len} to target (32k, 128k), rescale RoPE (linear interpolation or NTK-aware / YaRN), and train on filtered long-context documents (books, code repos, arXiv).
  • Adds ~5-10% compute for a huge context boost.
  • Alternative: 'position interpolation' fine-tuning on a small long-context set — cheap and effective for moderate extensions (2-8x).
#pretraining#long-contextPermalink & quiz →

Is data curriculum used in LLM pretraining?

hard
  • Yes but with restraint.
  • Common practice: start with a slightly noisier / broader mix, then finish the last 10-20% of training on a higher-quality curated 'annealing' set (e.g., Wikipedia, curated books, high-quality code, math) at a decayed learning rate.
  • LLaMA-3, Qwen-2 and Gemma explicitly use this 'quality upsampling at the end'.
  • Careful with curriculum: overly aggressive orderings can hurt generalization vs uniform sampling.
#pretraining#dataPermalink & quiz →

How much does it cost to pretrain a modern LLM?

hard
  • Rule of thumb: 6*N*T FLOPs, where N = parameters and T = training tokens.
  • LLaMA-3 8B on 15T tokens = ~7.2e23 FLOPs.
  • On an H100 doing ~1e15 FLOPs/s in bf16 (~50% MFU), that's ~1.5M GPU-hours (~3−5Matcloudprices).GPT−4estimatedat10−25MH100−hours( 3-5M at cloud prices). GPT-4 estimated at 10-25M H100-hours (~60-100M).
  • Frontier training is 10-100x more expensive than fine-tuning; inference at scale often dominates lifetime cost.
#pretraining#cost#productionPermalink & quiz →

Why is the softmax + cross-entropy at the LM head a training bottleneck?

hard
  • For a vocab V=128k and hidden d=4k, the logit matrix (batch × seq × V) can dominate activations — often the single biggest tensor in the graph.
  • Materializing it in fp32 wastes memory and compute.
  • Modern trick: 'fused CE' — compute logits and cross-entropy in a single fused CUDA kernel that never materializes the full V-dim tensor.
  • Liger Kernel and CUTCROSSENTROPY\mathrm{CUT}_{\mathrm{CROSS}}\mathrm{ENTROPY} (2024) do this — 2-5x memory reduction on the LM head, faster training.
#architecture#pretrainingPermalink & quiz →

What is continual pretraining and when does it beat fine-tuning?

hard
  • Take a pretrained base model and continue training on a large in-domain corpus (say, 10-100B tokens of medical literature, or code, or a specific language) with a reduced learning rate.
  • Result: a domain-adapted base model that can be further instruction-tuned.
  • Continual pretraining beats fine-tuning when: the domain corpus is large (>1B tokens), the target vocabulary/style differs from the base (medicine, law, non-English), or you need the model to know new facts baked in rather than retrieved.
#pretraining#fine-tuningPermalink & quiz →

What parallelism strategies are combined to train a 70B model?

hard
  • (1) Tensor parallelism (TP=4-8): split each matmul across GPUs on the same node using NVLink.
  • (2) Pipeline parallelism (PP=4-16): split layers across nodes, microbatch to hide bubbles.
  • (3) Data parallelism (DP=N): replicate across pipeline stages, each processes a different batch shard.
  • (4) ZeRO/FSDP: shard optimizer + gradients + parameters across DP replicas.
  • Total: 3D parallelism.
  • Add sequence parallelism for long context, expert parallelism for MoE.
  • Frameworks: Megatron-DeepSpeed, TorchTitan.
#pretraining#architecturePermalink & quiz →

What is embedding tying and when do LLMs use it?

medium
  • Share the same weight matrix between the input token embedding and the output LM head.
  • Saves ~V*d parameters (128k * 4k = 500M for a small LLM — huge share).
  • Slightly hurts quality on very large models (untied heads specialize output distribution).
  • Used in smaller LLMs (LLaMA-2 7B ties them; LLaMA-2 70B does not).
  • For very-large-vocab models it's often kept untied to preserve modeling flexibility.
#architecture#pretrainingPermalink & quiz →

What's the practical minimum tokens-per-parameter for a competitive LLM?

hard
  • Chinchilla optimal: ~20 tokens/param.
  • But 'compute-optimal at fixed budget' is not the same as 'best quality per param at fixed inference cost'.
  • Modern practice trains way past Chinchilla to get a smaller model with the same quality: LLaMA-3 8B on 15T tokens = 1875 tokens/param.
  • Rationale: inference cost dominates the model's lifetime; a smaller model that took more training compute is cheaper to serve.
#pretraining#scaling#costPermalink & quiz →

Can you repeat data across epochs in LLM pretraining?

hard
  • Yes, moderately — repeating a fixed pretraining set for 2-4 epochs is roughly equivalent to training on 2-4x more unique tokens on a log scale (Muennighoff 2023).
  • Beyond ~4 epochs, gains diminish sharply and memorization increases.
  • Data-constrained regimes (specialized languages, code, medicine) rely on this — modern practice mixes fresh data with high-quality repeated corpora, with repeat counts up to ~5-10 for the best subsets.
#pretraining#dataPermalink & quiz →

What is benchmark contamination and how do you detect it?

hard
  • The training corpus contains verbatim (or near-verbatim) copies of benchmark questions — the model memorizes answers and inflates eval scores.
  • Detection: (1) exact-string search for benchmark questions in the training set; (2) 'canary' strings — inject unique markers into curated benchmarks and check for regurgitation; (3) held-out contemporary benchmarks after model cutoff.
  • Modern LLM eval reports increasingly cite contamination-decontaminated scores.
#pretraining#evaluation#dataPermalink & quiz →

How is high-quality instruction-tuning data constructed?

medium
  • Sources: (1) hand-written by experts (expensive, small); (2) distilled from a stronger model with human filtering (OpenHermes, Alpaca-style — cheap, scalable); (3) real user conversations with filtering + rewriting; (4) task-mixture datasets (FLAN, T0).
  • Quality > quantity: LIMA (Meta, 2023) showed 1k carefully curated pairs beat 50k noisy ones.
  • Modern SFT recipes use 10k-100k high-quality examples across diverse tasks (chat, code, math, extraction, safety refusals).
#fine-tuning#data#alignmentPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz