Your anomaly detector has no labels. How do you set the decision threshold?
mediumAnswer
- Set it from operational capacity rather than from statistics.
- Decide how many alerts a human can genuinely review per day, then take that quantile of the score distribution, which turns an unanswerable question into a staffing one.
- Get a handful of confirmed cases, even a dozen from historical incidents, and check that they land above the threshold, because a threshold nothing known can pass is not a threshold.
- Track the score distribution over time, since drift shifts scores and a fixed absolute cutoff silently changes the alert volume.
- Then close the loop: every reviewed alert becomes a label, and after a few weeks you have enough to evaluate precision honestly and eventually train a supervised model.
Check yourself — multiple choice
- Use three standard deviations
- Derive it from review capacity as a score quantile, validate against known historical incidents, monitor for score drift, and turn reviewed alerts into labels
- Flag everything and filter later
- Thresholds are unnecessary
Alert capacity plus a few confirmed cases beats an arbitrary statistical cutoff, and review generates labels.
#anomaly-detection#evaluation
Practise Unsupervised Learning
214 interview questions in this topic.