How do regulators and banks validate market risk models through backtesting?

Backtesting is the primary tool for validating VaR models. It checks whether the model's confidence level matches the actual frequency of exceptions (days where the loss exceeds VaR).

The Basic Test:

For a 99% VaR model over 250 trading days, you expect approximately 2.5 exceptions (1% x 250). The question: is the observed number of exceptions consistent with a 1% expected rate?

Basel Traffic Light Framework:

Formal Statistical Tests:

1. Kupiec's Proportion of Failures (POF) Test:

Tests whether the observed exception rate equals the expected rate.

LR_POF = -2 x ln[(1-p)^(n-x) x p^x] + 2 x ln[(1-x/n)^(n-x) x (x/n)^x]

Where p = expected failure rate, x = observed exceptions, n = sample size.

Distributed chi-squared with 1 degree of freedom.

2. Christoffersen's Independence Test:

Tests whether exceptions are independent (not clustered). Clustered exceptions are a serious problem — they suggest the model fails to adapt to changing volatility.

3. Christoffersen's Conditional Coverage Test:

Combines the POF test and independence test:

LR_CC = LR_POF + LR_IND

Distributed chi-squared with 2 degrees of freedom.

Example — Foxworth Capital, 250-day backtest of 99% VaR:

Scenario A: 3 exceptions, evenly spaced -> Green zone, passes all tests

Scenario B: 3 exceptions, all in one week -> Green zone on POF, but FAILS independence test

Scenario C: 8 exceptions, spread across the year -> Yellow zone, fails POF test

Scenario B is particularly dangerous — it suggests the model doesn't capture volatility clustering.

Why Standard Backtesting Isn't Enough:

1. Low power: With 250 observations and 1% expected rate, it's hard to distinguish a 1% model from a 2% model
2. Point estimate only: VaR is either breached or not — the test doesn't consider how badly VaR was breached
3. No tail assessment: A $10M VaR breached by $11M and by $50M are treated identically

Basel FRTB Improvement:

The move from VaR to Expected Shortfall partly addresses limitation #3. ES considers the average loss in the tail, not just whether VaR was breached.

FRM Key Points:

• Know the traffic light zones and associated multipliers
• Understand that independence testing catches volatility clustering failures
• Conditional coverage test combines both frequency and independence
• ES backtesting is more complex (no standard regulatory framework yet)
• Too FEW exceptions can also be problematic — it may mean the model is too conservative, wasting capital

Practice backtesting problems in our FRM Part II question bank.