How does the gap statistic work?

hard

Answer

$W_{k}$
$\mathrm{Gap}(k)\; = \;E \cdot [\operatorname{log}\;W_{k} \cdot ]\; - \;\operatorname{log}$
$\mathrm{Gap}(k + 1)\; - \;s_{k + 1}$
$More principled than the elbow; handles cluster-vs-no-cluster case (returns k=1 when data is unstructured).$
$Costly (needs B ~ 20 reference simulations).$

Check yourself — multiple choice

Same as elbow
Compare log $W_{k}$ to expected under uniform null; pick smallest k with Gap(k) ≥ Gap(k+1) - s; handles k=1 case; costly
Random
Only tiny data

Gap statistic: log $W_{k}$ vs uniform null; principled k selection.

#clustering#evaluation

214 interview questions in this topic.