EasyDeepLearn
Unsupervised Learning · section 8 of 9

Topic modeling & text

10 interview questions on topic modeling & text, each answered in full. Free to read, no account needed.

What is topic modeling and when do you use LDA?

medium
  • Topic modeling discovers latent themes in a document corpus.
  • LDA (Latent Dirichlet Allocation) models each document as a mixture of topics and each topic as a distribution over words.
  • Use it for exploratory analysis of text collections without labels.
  • Modern alternatives: embedding-based topic modeling (BERTopic) — usually more coherent on short/noisy text.
#nlp#textPermalink & quiz →

TF-IDF weighting — formula and rationale.

easy
  • TF: term frequency in document (raw, log, or sublinear).
  • IDF: log⁡(N  /  dft)\operatorname{log}(N\; / \;\mathrm{df}_{t}) — downweights common words appearing in many documents.
  • Product TF-IDF highlights terms specific to a document.
  • Uses: bag-of-words features, BM25 (probabilistic IDF variant, dominant in classical IR).
  • Modern hybrid: BM25 + dense embeddings for retrieval → Elasticsearch does this natively (kNN + BM25 rrf).
#nlp#textPermalink & quiz →

BM25 — why is it still competitive with modern retrievers?

medium
  • Okapi BM25 = probabilistic TF-IDF with length normalization + tunable k1k_{1} (TF saturation) + b (length norm).
  • Very fast (inverted index), well-understood, competitive on exact-match / keyword-heavy queries where dense embeddings miss.
  • Modern practice: hybrid retrieval BM25 + dense embeddings + RRF (reciprocal rank fusion) = best-of-both.
  • Standard in Elasticsearch, Vespa, Qdrant hybrid mode, all modern RAG pipelines.
#nlp#text#applicationsPermalink & quiz →

Vector search / approximate nearest neighbors — main algorithms.

medium
  • (1) HNSW (Hierarchical Navigable Small World): graph-based, dominant in industry (Qdrant, Weaviate, pgvector).
  • (2) IVF-PQ (inverted file + product quantization): FAISS default for very large indexes.
  • (3) ScaNN (Google): partition + reranking.
  • (4) LSH (locality-sensitive hashing): older, still used for near-duplicate detection.
  • Trade-off: recall vs latency vs memory.
  • HNSW default for < 10M vectors; IVF-PQ for > 100M.
#nlp#applicationsPermalink & quiz →

How is RAG retrieval an unsupervised problem?

medium
  • RAG (retrieval-augmented generation) retrieves passages from a corpus based on unsupervised embeddings (no labels of 'relevant' pairs).
  • Uses: dense retrieval (sentence embeddings + cosine + ANN), often hybrid with BM25.
  • Fine-tuning the retriever is supervised (via query-doc labels), but out-of-the-box embeddings + cosine already give strong retrieval unsupervised.
  • Foundation of modern LLM-augmented systems.
#applications#nlpPermalink & quiz →

PMI and PPMI — what they measure.

hard
  • PMI(x, y) = log[P(x, y) / (P(x) P(y))].
  • Positive → co-occur more than random.
  • Negative → less than random.
  • PPMI = max(0, PMI) — drops negative values, more stable.
  • Uses: word association (Church & Hanks), building sparse word co-occurrence matrices before SVD (LSA); implicit target of Word2Vec (Levy & Goldberg).
  • Foundation of distributional semantics.
#nlp#theoryPermalink & quiz →

How do you cluster / retrieve code snippets?

hard
  • (1) Code-specific embeddings: CodeBERT, StarCoder, GTE-code, OpenAI text-embedding-3 handle code decently.
  • (2) Combine with AST features / API signatures.
  • (3) Cluster with HDBSCAN or embed + LSH for near-duplicate detection (deduplicate training data of code models).
  • (4) Retrieval-augmented: embed function bodies, retrieve similar for autocomplete.
  • Standard in GitHub Copilot's underlying retrieval, code deduplication for training.
#applications#nlpPermalink & quiz →

Interview: users complain search returns irrelevant results — how do you fix?

hard
  • (1) Analyze query logs: are they specific / broad / navigational / typos?
  • (2) Baseline: BM25 lexical.
  • Add: (3) Dense retrieval (sentence-transformers).
  • (4) Hybrid: BM25 + dense + RRF fusion.
  • (5) Cross-encoder reranker on top 100 → top 10.
  • (6) Query rewriting (HyDE / expansion for underspecified queries).
  • (7) Feedback logging (clicks) → LTR fine-tuning.
  • (8) A/B measure: click-through, dwell time, session success.
  • (9) Evaluate with human relevance judgments quarterly.
#interview#applications#nlpPermalink & quiz →

Interview: choose a sentence embedding model for a startup RAG.

medium
  • Constraints: quality vs cost vs latency vs vector dimension.
  • (1) OpenAI text-embedding-3-small: cheap ($0.02/1M), 1536 dim, hosted.
  • (2) Cohere embed-v3: multilingual, 1024 dim.
  • (3) Open-source: bge-small-en-v1.5 (384 dim, fast, high MTEB score), E5-large-v2 (1024 dim, top-tier).
  • (4) DIY fine-tuning: contrastive on domain pairs.
  • Rule: start with bge-small / OpenAI-3-small for speed; benchmark on your data (MTEB score alone is misleading).
#interview#applications#nlpPermalink & quiz →

Interview: 'you have 1M support tickets — how do you categorize them?'

hard
  • (1) Deduplicate near-identical tickets (LSH).
  • (2) Sentence-transformer embed (bge-small or multilingual if needed).
  • (3) UMAP to 20 dim.
  • (4) HDBSCAN → topic clusters.
  • (5) c-TF-IDF for cluster keywords → auto-label.
  • (6) Manual review of top 20 clusters + rename/merge/split with domain team.
  • (7) Train classifier on labeled clusters for future ticket routing.
  • (8) Monitor cluster drift monthly + retrain HDBSCAN if new topics emerge (novelty detection).
#interview#applications#nlpPermalink & quiz →

Practise Unsupervised Learning

All 9 sectionsFlashcardsQuiz