The elbow plot has no elbow. How do you pick k?
mediumAnswer
- Accept that a smooth curve is telling you the data has no crisp cluster count, which is common and not a failure of method.
- Look at the silhouette across a range of k and prefer a value that is locally best rather than globally optimal.
- Use the gap statistic, which compares within-cluster dispersion against a uniform reference and gives a principled comparison.
- Fit a Gaussian mixture and compare the Bayesian information criterion, which penalizes complexity explicitly.
- Then apply the constraint that usually decides it in practice: how many groups the organization can actually act on differently.
- Six segments a marketing team can staff beats a statistically optimal thirty-one nobody can use.
Check yourself — multiple choice
- Always pick k = 3
- A smooth curve means no crisp k: use silhouette, the gap statistic and BIC from a mixture model, then let actionability decide
- Increase k until inertia is zero
- Clustering is inapplicable
Multiple criteria plus operational actionability replace a nonexistent elbow.
#clustering#evaluation
Practise Unsupervised Learning
214 interview questions in this topic.