How do you select the decision threshold for a binary classifier?

medium

Answer

$\mathrm{FP}_{\mathrm{cost}} \cdot \mathrm{FP}\; + \;\mathrm{FN}_{\mathrm{cost}} \cdot \mathrm{FN}$
$Never tune threshold on the test set.$
$If probabilities are poorly calibrated, calibrate first, then threshold.$
$Threshold tuning is often more impactful than model choice.$

Check yourself — multiple choice

Always use 0.5
On validation, tune to your business objective (F1, fixed recall, expected cost)
Only on the training set
Threshold selection doesn't matter

Tune the threshold on validation using the actual downstream objective.

#metrics#classification#calibration#imbalance

215 interview questions in this topic.