What does 'emergence' mean in LLM capabilities and is it real?
hardAnswer
- Original claim (Wei et al., 2022): some capabilities are absent at small scale and appear sharply above a compute threshold — 'phase transitions'.
- Subsequent work (Schaeffer et al., 2023) showed many 'emergent' curves are artifacts of using discrete metrics (accuracy) instead of continuous ones (log-loss on the right sub-task); the underlying capability improves smoothly with scale.
- Real capabilities like in-context learning, chain-of-thought, and code do improve smoothly with scale — 'emergence' is more measurement than magic.
Check yourself — multiple choice
- Emergence is magic, always sharp
- Many 'sharp' emergent curves are artifacts of discrete metrics; capabilities usually improve smoothly with scale
- No such thing as scaling curves
- Emergence is only for text-to-image
Emergence: often a metric artifact; smooth improvement in log-loss on the underlying task.
#scaling#fundamentals
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is a large language model, in one paragraph?
- Walk through how one training step of an autoregressive LLM works.
- How does Byte-Pair Encoding (BPE) tokenization work?
- Why don't LLMs simply use character-level tokenization?
- Summarize Kaplan et al. (2020) scaling laws in one sentence.
- State the Chinchilla scaling insight and its practical impact.