How do you evaluate a classifier when FP and FN have different costs?

hard

Answer

$C_{\mathrm{FP}}$
$\mathrm{example}\; = \;C_{\mathrm{FP}}\; \cdot \;\mathrm{FPR}\; \cdot \;P(\mathrm{neg})\; + \;C_{\mathrm{FN}}\; \cdot \;\mathrm{FNR}\; \cdot \;P(\mathrm{pos})$
$Pick the threshold that minimizes it on validation.$
$Report expected loss in dollar / operational units, not just precision-recall.$
$If costs are unknown but ordinal (FN worse than FP), use ROC iso-cost lines: parallel lines whose slope encodes the FP/FN cost ratio — pick the operating point where the line first touches the ROC curve.$

Check yourself — multiple choice

Report only accuracy
Assign explicit FP/FN costs; minimize expected cost on validation and pick the threshold accordingly
Use only ROC-AUC
Cost-sensitive evaluation is not possible

Attach costs to FP and FN → minimize expected cost on validation → pick threshold.

#metrics#classification

215 interview questions in this topic.