How does vocabulary construction affect multilingual quality?
mediumAnswer
- A vocab trained on 90% English will tokenize French / Chinese / Arabic into many more tokens per character than English — so those languages see less context (fewer tokens fit in a fixed window), pay more per character, and have less semantic density per position.
- Modern multilingual LLMs balance the training mix across languages when building the BPE vocab, or use much bigger vocabs (200k+) to keep per-language rates similar.
- Byte-level BPE is a safety net — no OOV.
Check yourself — multiple choice
- Vocab is language-independent
- English-heavy vocab bloats non-English by 2-5× tokens/char → less context, more cost, worse density; multilingual vocabs balance the training mix
- Only Chinese matters
- BPE fixes everything
Vocab bloat: 2-5× tokens/char for underrepresented languages → mixed-language BPE training or larger vocabs.
#tokenization#pretraining
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- Walk through how one training step of an autoregressive LLM works.
- How does Byte-Pair Encoding (BPE) tokenization work?
- How does SentencePiece differ from BPE?
- Name three practical failure modes caused by tokenization.
- Why don't LLMs simply use character-level tokenization?
- What data goes into modern LLM pretraining?