How do you choose the decision threshold for a classifier in a business setting?
mediumAnswer
- Not at 0.5.
- Write down the cost of a false positive and the value of a true positive, then pick the threshold that maximizes expected value.
- If a false negative costs 10 times a false positive, the optimal threshold moves well below 0.5.
- When costs are unknown, work backwards from a capacity constraint: if the team can only call 500 customers a week, take the top 500 scores and report the precision you get there.
- Always choose the threshold on a validation set and confirm on a held-out set, because the optimum shifts with the base rate.
Check yourself — multiple choice
- Always 0.5
- Maximize expected value from explicit FP/FN costs, or work backwards from an operational capacity constraint
- Whatever maximizes accuracy
- The threshold does not matter
Thresholds follow from costs or capacity, never from the 0.5 default.
#metrics#metrics-classification
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Precision vs recall — when do you optimize each?
- When should you use PR-AUC instead of ROC-AUC?
- How do you read a confusion matrix and derive the key metrics?
- Why is accuracy a bad primary metric for many real-world problems?
- What is F-beta score and when is F1 not enough?
- Macro vs micro vs weighted averaging in multi-class metrics — how do you choose?