How does SentencePiece differ from BPE?
mediumAnswer
- SentencePiece (used by T5, PaLM, LLaMA-1) treats input as raw Unicode (spaces are represented as a special '▁' symbol) and supports two algorithms: BPE and Unigram LM.
- Language-agnostic — no pre-tokenization by whitespace, works for languages without spaces (Chinese, Japanese).
- Unigram trains a fixed-vocab model then prunes low-utility tokens.
- Same downstream effect as BPE; slightly different tokenization at edges.
Check yourself — multiple choice
- SentencePiece requires whitespace splitting
- Language-agnostic tokenizer (raw Unicode + ▁-space); supports BPE or Unigram — used by T5 / LLaMA
- Only for English
- Same as WordPiece
SentencePiece: raw-Unicode, whitespace-free tokenization — BPE or Unigram variants.
#tokenization
Practise LLMs & GenAI
214 interview questions in this topic.