How do I choose the optimal classification threshold for a financial model?

Question

AcadiFi · Accepted Answer

Optimal threshold depends on the decision's economic cost structure, not statistical metrics alone. Four common approaches:

1. Youden's J: maximize TPR - FPR. Finds point on ROC farthest from diagonal. Assumes equal costs.

2. Expected Cost Minimization: threshold* = argmin[FPR × N × C_FP + (1-TPR) × P × C_FN]. Requires cost estimates.

3. F1 or Fβ maximization: threshold that maximizes F1 on validation set. Useful for imbalanced data.

4. Top-K: pick threshold that flags exactly K cases to match operational capacity.

Example: Chambersworth Capital's loan approval model produces probabilities of default. Dataset has 14,000 apps, 3% default rate.

Cost analysis: default costs $68,000/loan; declining a good loan loses $4,500 profit.

At threshold 0.04: approves 12,600 loans. 320 defaults × $68K = $21.8M losses, 500 good declined × $4.5K = $2.25M lost profit. Total cost $24M.

At threshold 0.08: approves 11,400 loans. 180 defaults × $68K = $12.2M, 1,180 good declined × $4.5K = $5.3M. Total $17.5M.

At threshold 0.12: approves 10,200 loans. 105 defaults × $68K = $7.1M, 2,020 good declined × $4.5K = $9.1M. Total $16.2M.

Sweep shows optimum near 0.13 (total $15.8M). Chambersworth adopts 0.13 as production threshold but monitors monthly — economic conditions shift default base rate, and they re-calibrate quarterly. CFA curriculum: always document cost assumptions and run sensitivity analysis on ±20% cost shifts.

How do I choose the optimal classification threshold for a financial model?

Master Level II with our CFA Course

Related Questions

Practice Questions