How does Byte-Pair Encoding (BPE) tokenization work?
mediumAnswer
- Start with a vocabulary of individual bytes/chars.
- Repeatedly find the most frequent adjacent pair in the corpus and merge it into a new token, until the desired vocab size (e.g., 32k, 100k, 200k).
- At inference, apply the same merge rules greedily.
- Byte-level BPE (GPT-2+, Llama, Mistral) starts from raw bytes → handles any Unicode without OOV.
- Common vocab sizes: 32k (Llama-2), 100k (GPT-4), 200k+ (Gemini).
- Larger vocab = fewer tokens per text = more efficient long-context inference.
Check yourself — multiple choice
- Tokenize by fixed-length substrings
- Greedily merge the most frequent byte pairs until reaching target vocab size; byte-level handles any Unicode
- BPE requires word-level split first
- BPE has fixed 512 vocab
BPE: iteratively merge frequent byte pairs; byte-level BPE avoids OOV entirely.
#tokenization#fundamentals
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Why don't LLMs simply use character-level tokenization?
- What is a large language model, in one paragraph?
- Walk through how one training step of an autoregressive LLM works.
- How does SentencePiece differ from BPE?
- Name three practical failure modes caused by tokenization.
- What does 'emergence' mean in LLM capabilities and is it real?