What quantitative techniques are used in model validation?

Question

AcadiFi · Accepted Answer

Quantitative validation techniques fall into five families:

1. Replication. Independent re-implementation of the model in a different environment (Python vs R, or different library). Confirms specification is correctly coded. Usually done for Tier 1 models. Discrepancies trigger root-cause analysis.

2. Sensitivity analysis. Perturb inputs one at a time and observe output changes. Does a 10% increase in default rate flow through to expected loss proportionally? Identifies non-linearities and unexpected interactions. Critical for interest rate and market risk models.

3. Stress testing. Run extreme but plausible scenarios through the model. Does it remain numerically stable? For copula credit models: run with asset correlations at 0.9 to check tail behavior. For option models: extreme vol, near-zero time-to-maturity, zero strike.

4. Benchmarking. As discussed — alternative models, standardized approaches, industry consensus. Reveals specification-level bias.

5. Backtesting / outcomes analysis. As discussed — compare to realized outcomes.

Additional specialized techniques:

- Parameter stability. Re-estimate the model on rolling windows; do coefficients drift?
- Bootstrap confidence intervals. Quantify estimation uncertainty.
- Ablation. Drop each feature and measure performance loss.
- Adversarial testing. Construct input sets designed to break the model (important for ML models subject to FRTB or EBA guidelines).

A concrete example: Sandspur Financial validates their new Heston option-pricing model. Replication in QuantLib confirms FFT pricing matches within 0.01 vol points. Sensitivity shows vega and volga behave smoothly. Stress testing finds numerical instability when kappa*theta/xi^2 drops below 0.3 (near Feller violation) — they add a safeguard. Benchmarking against local vol shows expected forward-skew divergence. Outcomes analysis on 6 months of P&L attribution shows the model captures vega/volga/vanna well.

For FRM exam: know that replication, sensitivity, stress, benchmarking, and outcomes are the five legs. A validation missing any is incomplete.

What quantitative techniques are used in model validation?

Master Part II with our FRM Course

Related Questions

Practice Questions