14 interview questions on tokenization & embeddings, each answered in full. Free to read, no account needed.
What are text embeddings and how are they used?
easy- Embeddings map text into a dense vector space where semantic similarity corresponds to geometric proximity (cosine or dot product).
- They are produced by encoders trained with contrastive or masked objectives.
- Uses: semantic search, RAG retrieval, clustering, deduplication, classification, and recommendation.
- Choose a model that matches your language, domain, and length; measure quality on your own retrieval eval set.
How does SentencePiece differ from BPE?
medium- SentencePiece (used by T5, PaLM, LLaMA-1) treats input as raw Unicode (spaces are represented as a special '▁' symbol) and supports two algorithms: BPE and Unigram LM.
- Language-agnostic — no pre-tokenization by whitespace, works for languages without spaces (Chinese, Japanese).
- Unigram trains a fixed-vocab model then prunes low-utility tokens.
- Same downstream effect as BPE; slightly different tokenization at edges.
Name three practical failure modes caused by tokenization.
medium- (1) Number-splitting: '1234' can tokenize into ['12', '34'] → poor arithmetic.
- Fix: digit-level tokenizers or explicit tools.
- (2) Code / whitespace fragility: leading spaces change tokens (hellovshello), breaking few-shot examples.
- (3) Non-English bloat: languages without English-heavy training data use 2-5x more tokens per character → higher cost and shorter effective context.
- (4) Rare characters produce multiple bytes each, wasting context.
How does vocabulary construction affect multilingual quality?
medium- A vocab trained on 90% English will tokenize French / Chinese / Arabic into many more tokens per character than English — so those languages see less context (fewer tokens fit in a fixed window), pay more per character, and have less semantic density per position.
- Modern multilingual LLMs balance the training mix across languages when building the BPE vocab, or use much bigger vocabs (200k+) to keep per-language rates similar.
- Byte-level BPE is a safety net — no OOV.
How do you pick an embedding model for a new RAG project?
medium- Criteria: (1) MTEB leaderboard score for your task type (retrieval, STS, classification); (2) language coverage — E5, BGE, Cohere multilingual for non-English; (3) max input length — most cap at 512 tokens; longer models (Jina, Nomic, BGE-M3) go 8k+; (4) dimension — 768/1024 typical; 3072 for OpenAI text-embedding-3-large; (5) cost + license.
- Modern strong choices: BGE-M3, E5-mistral, Nomic v1.5, Cohere v3, OpenAI text-embedding-3.
- Always validate on your own retrieval eval set — MTEB is not your domain.
What's the trade-off between embedding dimension and retrieval quality?
medium- Higher dim = more expressive space, marginally better recall on hard queries, more storage, slower ANN search.
- E.g., OpenAI text-embedding-3-large supports 3072 dim but can be shortened to 256/512/1024 via Matryoshka Representation Learning (MRL) — a training trick that keeps quality high at lower dims.
- Practical: 768-1024 is the sweet spot for most RAG; go higher only if benchmarks justify it.
- Storage matters at scale: 3072-dim × 4 bytes × 10M vectors = 120 GB.
What is Matryoshka Representation Learning (MRL)?
hard- Train the embedding model with a loss that operates at multiple truncation dims simultaneously (256, 512, 1024, 2048) — the first k dimensions must independently be a good embedding for any k.
- At inference, you can slice to any size without retraining.
- Enables cheap-quality trade-offs: index at 1024 for search, truncate to 256 for a 4x smaller cache or a faster reranker input.
- Adopted by OpenAI, Cohere, Nomic, Jina.
How is ColBERT different from a standard bi-encoder?
hard- Bi-encoder: encode query → 1 vector, encode doc → 1 vector, score = cosine (single interaction).
- ColBERT (Khattab 2020): encode query → many token vectors, encode doc → many token vectors.
- Score = sum over query tokens of max cosine to any doc token ('MaxSim' late interaction).
- Higher recall than bi-encoders because tokens can match individually, cheaper than cross-encoders at retrieval time (docs are still pre-indexed).
- ColBERT-v2 uses PQ to compress token vectors → practical at scale.
What are 'contextual embeddings' and why are they useful?
hard- Anthropic's Contextual Retrieval (2024): before embedding a chunk, prepend a short LLM-generated context that situates the chunk within its parent document ('This chunk is from a Q3 2024 earnings report, discussing revenue growth in EMEA').
- Embed the contextualized chunk.
- Solves the 'orphan chunk' problem where standalone chunks lose important framing. 35% retrieval error reduction, up to 67% when combined with reranking.
- Cost: one LLM call per chunk at index time — bearable with prompt caching.
How does multimodal retrieval (image + text) work?
hard- Use a shared-embedding model like CLIP (Radford 2021), SigLIP (2023), Jina CLIP, or Cohere Embed v3-Image.
- These map images and text into a common vector space via contrastive training.
- At retrieval time, embed a text query, do ANN search over image embeddings (or vice versa).
- Applications: image search from text, semantic album search, cross-modal RAG (retrieve images relevant to a question).
- Combine with a multimodal LLM (GPT-4o, Claude, Gemini) for downstream generation over retrieved images.
When and how do you fine-tune an embedding model?
hard- Fine-tune when off-the-shelf embeddings underperform on your domain (specialized jargon, product names, code, non-English).
- Method: (1) collect (query, positive-doc) pairs from usage logs; (2) contrastive training (info-NCE loss) with in-batch negatives or hard negatives mined from a base retriever; (3) start from a strong base (E5, BGE, Nomic) — a few thousand pairs is often enough for LoRA-style fine-tuning.
- Gains: 5-20% recall@10 over base.
- Watch out for retraining bias — always eval on a held-out real-query set.
What are 'hard negatives' in embedding training?
hard- Negatives that look superficially similar to the positive but are actually irrelevant — force the model to learn fine-grained distinctions.
- Mining strategies: (1) BM25 top-k excluding the positive (fast, common); (2) previous-generation retriever's top-k excluding the positive (harder); (3) LLM-generated distractors ('write a passage that's topically similar but doesn't answer the query').
- Better than random in-batch negatives, which are almost always easy.
- Standard in modern embedding training (E5, BGE, GTE).
How does CLIP enable multimodal capability?
medium- CLIP (Radford 2021): trained a text encoder and image encoder jointly with contrastive loss on 400M web (image, caption) pairs.
- Aligns them in a shared embedding space — same-semantic image and caption have similar embeddings.
- Enables: zero-shot image classification (embed candidate labels, pick nearest), image search from text, cross-modal retrieval.
- Foundation for later multimodal LLMs — most VLMs use a CLIP-style encoder to embed images before passing to the transformer.
What breaks when you change the embedding model in a live RAG system?
hard- Everything in the index, because embeddings from different models are not comparable: vectors live in different spaces, so mixing them silently produces nonsense similarities.
- You must re-embed the entire corpus and rebuild the index, which costs money and time proportional to the corpus.
- Dimensionality often changes, so the store's schema and any dimensionality assumptions change too.
- Retrieval quality can shift in both directions, so you need your retrieval evaluation set before the swap to compare.
- The safe pattern is to build the new index alongside the old one, compare on the eval set, then cut over atomically.