Why is benchmarking against alternative models part of validation?

Question

AcadiFi · Accepted Answer

Benchmarking — comparing the production model against one or more alternative models — is a core component of conceptual soundness review under SR 11-7. It tests whether the chosen specification is among the best, not merely adequate.

Three purposes:

1. Challenge specification choices. If a bank uses logistic regression for PD, a challenger might be a gradient-boosted tree or a random forest. Large performance differences (e.g., challenger Gini materially higher) suggest the production model is suboptimal or misspecified.

2. Reveal hidden assumptions. Comparing a structural credit model (Merton-KMV) to a reduced-form model (Jarrow-Turnbull) surfaces assumptions about asset-volatility observability and jump risk.

3. Stress-test interpretability. A challenger with similar performance but different feature importances hints at unstable feature attribution.

Benchmarking doesn't require adoption of the challenger. It requires understanding why they differ and documenting the decision.

A concrete example: Northfork National Bank's C&I PD model is a logistic regression with 12 variables. The validation team built a gradient-boosted tree challenger on the same development sample. The tree achieved Gini 0.74 vs logistic's 0.68. Investigation showed the tree exploited a non-linear interaction between 'revenue growth' and 'leverage' that logistic missed. Validation recommended adding an interaction term to production, lifting its Gini to 0.72 and closing the gap.

For capital models, Basel IMA requires banks to benchmark their internal VaR against a standardized approach (SA-CCR, FRTB SA) at least annually. Large divergences require investigation and may lead to multiplier add-ons.

For FRM Part II: benchmarking is distinct from backtesting. Backtesting compares model predictions to outcomes; benchmarking compares model A's predictions to model B's. Both are required components of a robust validation.

Why is benchmarking against alternative models part of validation?

Master Part II with our FRM Course

Related Questions

Practice Questions