Is it a good idea to run k-means on raw text embeddings?

hard

Answer

$It works, with caveats worth stating.$
$Embeddings are high-dimensional and roughly isotropic, so Euclidean distances concentrate and everything looks similarly far apart, which weakens the contrast k-means depends on.$
$Use cosine similarity instead, which for normalized vectors is equivalent to Euclidean distance on the sphere, so normalizing your vectors and then running k-means is the usual fix.$
$Reducing to a few dozen dimensions with PCA or UMAP before clustering often improves both quality and speed.$
$Also remember that embeddings cluster by whatever the encoder was trained to emphasize, frequently topic and register, so if you need a different notion of similarity no clustering algorithm will recover it.$

Check yourself — multiple choice

Yes, with no adjustment
Normalize first so k-means operates on cosine geometry, reduce dimensionality before clustering, and accept that clusters reflect whatever similarity the encoder learned
Never cluster embeddings
Only hierarchical clustering works

Normalization, dimensionality reduction and awareness of the encoder's notion of similarity make it work.

#clustering#representation-learning#nlp

214 interview questions in this topic.