Name three practical failure modes caused by tokenization.
mediumAnswer
- (1) Number-splitting: '1234' can tokenize into ['12', '34'] → poor arithmetic.
- Fix: digit-level tokenizers or explicit tools.
- (2) Code / whitespace fragility: leading spaces change tokens , breaking few-shot examples.
- (3) Non-English bloat: languages without English-heavy training data use 2-5x more tokens per character → higher cost and shorter effective context.
- (4) Rare characters produce multiple bytes each, wasting context.
Check yourself — multiple choice
- Tokenization has no failure modes
- Number splitting, whitespace fragility, non-English token bloat, rare-char explosion
- Only affects Chinese
- Solved by higher temperature
Real-world tokenization pain: numbers, whitespace, non-English bloat, rare chars.
#tokenization
Practise LLMs & GenAI
214 interview questions in this topic.