What should ongoing model monitoring look like between validations?

Question

AcadiFi · Accepted Answer

Ongoing monitoring (OGM) is the between-validation surveillance that detects model deterioration before it causes material risk. Three tracks:

1. Performance monitoring. Direct comparison of model outputs to realized outcomes. For PD models: Kolmogorov-Smirnov, Gini, Hosmer-Lemeshow tested quarterly. For VaR: Kupiec and Christoffersen backtests on exception counts. Thresholds must be pre-defined (e.g., Gini drop >10% triggers amber; >20% triggers red).

2. Input stability. Are input variables still within the development sample's distribution? Population Stability Index (PSI) below 0.10 is stable; 0.10-0.25 is moderate drift; above 0.25 signals significant shift requiring investigation.

3. Process monitoring. Are production runs completing successfully? Are upstream data feeds on time and complete? Any overrides by users?

A monitoring framework distinguishes three response tiers:

- Green: continue monitoring.
- Amber: root-cause analysis required within 30 days. May impose conservative overlay.
- Red: immediate escalation to MRM committee. Possible model suspension or recalibration.

A concrete example: Linwood Capital Bank monitors their retail PD model monthly. In Q2 2025 the PSI on 'debt-to-income' jumped from 0.08 to 0.19 after a new origination channel launched. Gini dropped from 0.68 to 0.61. The validation team triggered an amber review, found the new channel had different borrower demographics, recalibrated the segment-specific intercept, and cleared the finding. Without OGM, the model would have silently under-reserved by an estimated $45M over the year.

For high-frequency models (algo trading, fraud), OGM runs intraday with automatic circuit breakers. For slow-moving models (CCAR loss projections), quarterly OGM suffices. FRM exam: OGM is mandatory between full validations and is the most common source of early-warning findings.

What should ongoing model monitoring look like between validations?

Master Part II with our FRM Course

Related Questions

Practice Questions