Why don't LLMs simply use character-level tokenization?

medium

Answer

$O(n^{2})$
$Also, characters carry little semantic information — each attention step handles less content, requiring deeper / wider models.$
$BPE / SentencePiece with 32k-200k vocab balances sequence length vs semantic density.$
$Byte-level BPE gives the character-level robustness (any Unicode) without the sequence-length penalty.$

Check yourself — multiple choice

Character-level is more expressive
Sequences would be 5-10× longer → $O(n^{2})$ attention becomes too expensive; also less semantic density per token
Characters have too few tokens
Only used for Chinese

Char-level: sequences too long for $O(n^{2})$ attention; BPE trades vocab size for compact sequences.

#tokenization#fundamentals

214 interview questions in this topic.