LDA or embedding-based topic modelling for a corpus of support tickets?

medium

Answer

$Embedding-based clustering is usually the better modern default.$
$LDA treats a document as a bag of words with no notion of synonymy, which hurts badly on short texts like tickets, where there are too few words per document to estimate a topic mixture reliably.$
$Embedding the tickets, reducing dimensionality and clustering, then labelling each cluster by its most distinctive terms, handles paraphrase and short text far better.$
$LDA still has advantages: it gives each document a genuine mixture over topics rather than one assignment, it is cheap, and its probabilistic form is easier to defend statistically.$
$Whichever you pick, topic quality is judged by whether humans can name the topics, so run a coherence measure and then read samples.$

Check yourself — multiple choice

LDA is always superior
Embedding-based clustering handles short text and synonymy better; LDA keeps genuine per-document mixtures and cheapness — judge either by human-nameable topics and coherence
Neither works on tickets
Only supervised classification works

Short documents defeat bag-of-words mixtures, but LDA retains mixture semantics and low cost.

#nlp#text#clustering

214 interview questions in this topic.