EasyDeepLearn
Unsupervised Learning · section 4 of 9

Cluster evaluation

5 interview questions on cluster evaluation, each answered in full. Free to read, no account needed.

How do you evaluate anomaly detection?

medium
  • Usually severely imbalanced → don't use accuracy.
  • Standard metrics: (1) ROC AUC (rank-based, threshold-free), (2) PR AUC (better under extreme imbalance), (3) precision@k and recall@k at operating threshold, (4) F1 for a chosen threshold.
  • Cost-sensitive: business cost of false positive / false negative.
  • Also: alert precision and MTTR (time to detect) in production ops contexts.
#anomaly-detection#evaluationPermalink & quiz →

How do you evaluate the quality of a self-supervised representation?

medium
  • (1) Linear probing: freeze encoder, train linear classifier on downstream labels → measures 'linearly separable info'.
  • (2) Fine-tuning: unfreeze, full downstream training → measures 'total info'.
  • (3) k-NN classification in embedding space → non-parametric.
  • (4) Transfer to many tasks (VTAB, ELEVATER).
  • (5) Robustness under distribution shift.
  • Rule: report linear probe + k-NN + fine-tune; don't over-index on one downstream task.
#representation-learning#evaluationPermalink & quiz →

How do you evaluate topic quality?

medium
  • (1) Coherence metrics: CVC_{V}, CUMassC_{\mathrm{UMass}}, CNPMIC_{\mathrm{NPMI}} (Röder et al.) — measure semantic similarity of top-k topic words.
  • (2) Topic diversity: fraction of unique top-k words across topics.
  • (3) Downstream utility: classification accuracy using topic features.
  • (4) Human evaluation: 'word intrusion' task.
  • (5) Perplexity on held-out — surprisingly poor predictor of human-judged quality.
  • Rule: report coherence + diversity + qualitative inspection.
#nlp#text#evaluationPermalink & quiz →

Your anomaly detector has no labels. How do you set the decision threshold?

medium
  • Set it from operational capacity rather than from statistics.
  • Decide how many alerts a human can genuinely review per day, then take that quantile of the score distribution, which turns an unanswerable question into a staffing one.
  • Get a handful of confirmed cases, even a dozen from historical incidents, and check that they land above the threshold, because a threshold nothing known can pass is not a threshold.
  • Track the score distribution over time, since drift shifts scores and a fixed absolute cutoff silently changes the alert volume.
  • Then close the loop: every reviewed alert becomes a label, and after a few weeks you have enough to evaluate precision honestly and eventually train a supervised model.
#anomaly-detection#evaluationPermalink & quiz →

What makes a good self-supervised pretext task?

hard
  • It must be solvable only by learning something you actually want.
  • The failure mode is a shortcut: a task the network can solve with a superficial cue, such as detecting a rotation from a border artefact or matching two crops by their shared colour histogram, which yields a high pretext score and useless representations.
  • Good tasks force invariance to nuisance factors while preserving the semantics you need downstream, which is why contrastive augmentation design matters more than the loss function.
  • The task also has to be hard enough to require capacity but not so ambiguous that the target is unpredictable in principle.
  • The only real test is transfer: freeze the encoder and measure linear-probe performance on the downstream task, because pretext loss is not the objective you care about.
#representation-learning#evaluationPermalink & quiz →

Practise Unsupervised Learning

All 9 sectionsFlashcardsQuiz