Walk through how one training step of an autoregressive LLM works.
easyAnswer
- Take a text batch, tokenize, forward the transformer to get logits for every position (predicting next token from each prefix).
- Compute cross-entropy against the true next tokens with a causal mask so position t only sees .
- Sum losses over all positions (that's L-1 predictions per L-token sequence — huge signal per step, much cheaper than one prediction per sequence).
- Backprop, AdamW step.
- Repeat on trillions of tokens.
Check yourself — multiple choice
- Only the last token is predicted per sequence
- Every position predicts its next token under a causal mask → L-1 gradients per L-token sequence
- LLMs use bidirectional attention during training
- Cross-entropy is not used
Causal LM training: predict every next token with a causal mask — L-1 gradients per sequence.
#fundamentals#pretraining
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is a large language model, in one paragraph?
- How does Byte-Pair Encoding (BPE) tokenization work?
- Why don't LLMs simply use character-level tokenization?
- What data goes into modern LLM pretraining?
- Why is data deduplication critical for LLM pretraining?
- Summarize Kaplan et al. (2020) scaling laws in one sentence.