Backtesting is a general and widely used procedure for evaluating model performance for certain types of financial models.
Backtesting is similar to an out-of-sample (cross-) validation process, but differs in that it is an ongoing exercise (post model deployment) rather than a step in model development or validation process.
Backtesting compares the latest set of model predictions against actual realizations, with the validation sample being formed by an independent segment of data.
Preconditions for using backtesting:
- The model being backtested is sufficiently well understood that an estimate fail rate can be constructed
- There substantial data generated for backtesting purposes so as to enable deriving statistics
Backtesting market risk VaR is a long-time regulatory requirement and well developed practice. Daily VaR estimates are compared with the ex-post PnL realizations. Instances where the realization exceeds the estimate are denoted exceptions. Assuming independence of market shocks over time for any given VaR system there is a certain expected frequency of such exceptions.
Issues and Challenges
- Out of the three major risk types (market, credit, operational risk) only market data and certain areas of retail credit and high frequency operational events can provide reliable backtesting datasets (sufficient number of new observations).
- Backtesting is a historical (backward) looking validation process and successful tests do not guarantee future performance
- BCBS 22: Supervisory framework for the use of “backtesting” in conjunction with the internal models approach to market risk capital requirements