How do you evaluate topic quality?
mediumAnswer
- (1) Coherence metrics: , , (Röder et al.) — measure semantic similarity of top-k topic words.
- (2) Topic diversity: fraction of unique top-k words across topics.
- (3) Downstream utility: classification accuracy using topic features.
- (4) Human evaluation: 'word intrusion' task.
- (5) Perplexity on held-out — surprisingly poor predictor of human-judged quality.
- Rule: report coherence + diversity + qualitative inspection.
Check yourself — multiple choice
- Perplexity
- Coherence utility + word-intrusion humans; perplexity poor predictor — never rely on it alone
- Random
- Only silhouette
Topic eval: coherence + diversity + downstream + intrusion; not perplexity.
#nlp#text#evaluation
Practise Unsupervised Learning
214 interview questions in this topic.