Backtesting
Definition
Backtesting is a general and widely used procedure for evaluating model performance for certain types of financial models. It is an ex-post comparison of observed outcomes with expected outcomes derived from the use of a model.
Usage
Backtesting is similar to an out-of-sample (cross-) validation process, but differs in that it is an ongoing exercise (post model deployment) rather than a step in model development or validation process. Backtesting compares the latest set of model predictions against actual realizations, with the validation sample being formed by an independent segment of data.
Preconditions for using backtesting:
- The model being backtested is sufficiently well understood that an estimated fail rate can be constructed
- There is substantial data generated for backtesting purposes so as to enable deriving statistics
Out of the three major regulatory risk types (Market Risk, Credit Risk, Operational Risk) only market data and certain areas of retail credit and high frequency operational risk events can provide reliable backtesting datasets (sufficient number of new observations).
Example
Backtesting market risk VaR is a long-time regulatory requirement[1] and well developed practice. Daily VaR estimates are compared with the ex-post PnL realizations. Instances where the realization exceeds the estimate are denoted exceptions. Assuming independence of market shocks over time for any given VaR system there is a certain expected frequency of such exceptions.
Issues and Challenges
- Backtesting is a historical (backward) looking validation process and successful tests do not guarantee future performance
- Backtests that focus on the number of VaR violations have low power when the number of VaR exceptions is small. The power of backtests can be improved modestly through the use of conditional backtests or other techniques that consider multiple dimensions of the data like the timing of violations or the magnitude of the VaR exceptions. No consensus has yet emerged on the relative benefits of using actual or hypothetical results (ie P&L) to conduct backtesting exercises.[2]
