What is in-context learning (ICL) and how does it work?
mediumAnswer
- Ability of LLMs to solve new tasks from examples provided in the prompt (few-shot) without weight updates.
- E.g., show 3 (English, French) pairs and ask for a fourth translation.
- The model performs 'meta-learning' during pretraining: it saw enough patterns to recognize a new task's structure from a handful of examples.
- Mechanistic interpretation: attention heads implement an approximate gradient descent step over the in-context examples ('learning without weight updates' — Akyürek et al. 2022).
Check yourself — multiple choice
- ICL updates weights
- Solve new task from prompt-provided examples without weight updates; mechanistically ≈ gradient step in attention over examples
- Only zero-shot exists
- ICL requires SFT
ICL: few-shot examples in the prompt trigger 'meta-learned' pattern completion.
#prompting#in-context-learning#fundamentals
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Few-shot vs zero-shot — when do you use each?
- What is a large language model, in one paragraph?
- What are the pillars of good prompt engineering?
- Walk through how one training step of an autoregressive LLM works.
- How does Byte-Pair Encoding (BPE) tokenization work?
- Why don't LLMs simply use character-level tokenization?