What is model risk, and how do banks validate their risk models to avoid catastrophic failures?

Model risk arises when a model used for decision-making produces incorrect or misleading outputs. After the 2008 crisis exposed massive model failures — CDO pricing models, VaR models, stress testing models — regulators mandated formal model risk management frameworks. **Sources of Model Risk:** **1. Specification Error (Wrong Model)** - Using a normal distribution when returns are fat-tailed - Assuming constant correlations when they spike in crises - Example: Fairview Capital's VaR model assumed log-normal returns for structured credit. During the crisis, correlation between mortgage tranches jumped from 0.3 to 0.9, and the model underestimated losses by $420 million. **2. Implementation Error (Right Model, Wrong Code)** - Programming bugs, data feed errors, incorrect parameter mapping - Example: A quant at Osprey Trading implements a Black-Scholes model but accidentally uses calendar days instead of trading days for the time parameter, inflating option values by ~15%. **3. Calibration Error (Right Model, Wrong Parameters)** - Using stale parameters, insufficient historical data, or survivorship-biased samples - Example: Calibrating a GARCH model with only 2019-2021 data (including the COVID crash) produces volatility estimates biased upward for normal markets. **4. Application Error (Right Model, Wrong Use)** - Using a model outside its intended scope or for decisions it wasn't designed to support - Example: Using a short-term VaR model calibrated for liquid equities to estimate risk on illiquid private credit positions. **The Model Validation Framework (SR 11-7):** The Federal Reserve's SR 11-7 guidance establishes three lines of defense: **First Line:** Model developers (quants who build and calibrate) **Second Line:** Independent model validation team (challenges assumptions and tests independently) **Third Line:** Internal audit (reviews the overall framework) **Key Validation Activities:** **A. Conceptual Soundness Review** - Is the mathematical framework appropriate for the product/risk being modeled? - Are the assumptions reasonable and well-documented? - Review published literature — is there academic support for the approach? **B. Backtesting** Compare model predictions to actual outcomes: **VaR Backtesting Example:** Overton Risk Analytics runs a 99% 1-day VaR model over 250 trading days. At 99% confidence, you'd expect ~2.5 exceptions (days where losses exceed VaR). | Zone | Exceptions | Conclusion | |---|---|---| | Green | 0-4 | Model is acceptable | | Yellow | 5-9 | Investigate further | | Red | 10+ | Model is likely flawed, capital surcharge applied | If Overton observes 12 exceptions in 250 days, the model is in the red zone — it's systematically underestimating risk. **C. Benchmarking** - Compare model output against alternative models or industry standards - If your proprietary model says a CDO tranche is worth $82M but Bloomberg's model says $71M, the $11M gap needs explanation **D. Sensitivity Analysis** - Stress individual inputs to understand model behavior at extremes - How does VaR change if you double the assumed correlation? If volatility triples? - Identify which inputs the model is most sensitive to **E. Outcomes Analysis** - Track model performance over time (P&L attribution, actual vs. predicted defaults) - Deteriorating performance triggers model review or retirement **Model Risk Mitigation:** 1. **Model inventory:** Maintain a complete registry of all models with risk ratings (Tier 1/2/3) 2. **Annual review cycle:** High-risk models validated annually, lower-risk on 2-3 year cycles 3. **Model limits:** Set usage boundaries — "this model is valid for tenors up to 5 years and notional up to $500M" 4. **Champion-challenger:** Run the production model alongside an alternative to continuously benchmark **Exam Tip:** FRM Part II frequently tests the SR 11-7 framework, the Basel traffic light approach to VaR backtesting, and the ability to identify which source of model risk a scenario describes. Know the three lines of defense cold. Practice model risk scenarios in our FRM Part II question bank.