Model Risk

From Open Risk Manual

Definition

Model Risk refers to the potential for error in the development, implementation and/or the application or interpretation of results produced of a financial / Risk Model. Model risk related errors are risk factors that can lead to a variety of financial and / or reputational Loss events. Model risk is generally considered to be a type of Operational Risk.

Context

Financial / risk models are mathematical representation of future states of the world along with the description of those states in financial / economic terms. All models of this category are thus essentially forecasting tools that project explicit or implicit scenarios about the future evolution of economic or financial variables. When models are applied to the assessment of risk, which is a common use in financial services, model risk can also be seen as a second-order risk (a contributing factor to the risk of poor risk management)

Causes and Manifestations

A significant factor in model risk is the organization, skills and performance of technically trained individuals (modelers, data scientists, quants etc.) who have the mandate to develop and implement models. Such individuals maybe either in the employment of the firm or external (e.g., consultants, employed by rating agencies or other analytics firms). In the most abstract sense, modelers are tasked to create a mathematical representation of future states of the world on the basis of past and current information ("the model"). Models are then used in a variety of contexts to assist with decision making, including Risk Acceptance, pricing, Risk Management and a variety of other business activities.

Model risk enters all operational aspects of the model development and use process in the form of

  • Limited ability and/or willingness of technical personnel to create the best possible, error free, model, caused for example by:
    • Unclear mandate / objective or scope of the model
    • Poor or missing data (which can refer to both historical data and data relevant for capturing the current state)
    • Flawed reasoning (in the form of invalid assumptions or incoherent logical structure)
    • A variety of possible biases that may be reflected in model structure
    • Erroneous application of mathematics
    • Pretence of Knowledge or Spurious Accuracy
  • Intrinsic Model Risk: the multiplicity of possible models consistent with the same factual base / ground truth
  • Limited ability and/or willingness of the organization to deploy and use models is appropriate way
    • Inappropriate application (scope)
    • Inadequate IT infrastructure
    • Inadequate monitoring of model performance

Underlying Factors

  • Ultimately, the most challenging forms of model risk originate from deeper difficulty of representing complex organizational financial or economic activity using relatively simple (tractable) recipes.
  • An organization's ability to manage model risk may be limited by resources (personnel, data, knowledge, time, computational facilities etc.)
  • The willingness of the modeler may also be affected by the specific financial, reputational and other elements of his/her role, which may provide adverse incentives and generate biases.
  • The "modelability" of the problem (See Quantifiable versus Unquantifiable Risks)

Comparison with other risk types

Model risk is analogous to other risk types in the sense that it can lead to real and unanticipated financial and reputational losses. For example:

  • a faulty credit scorecard can lead to business being accepted that will lead to unexpectedly large credit losses (which may have not been priced-in)
  • a faulty derivatives pricing model can lead to pricing and risk management decisions that lead to significant losses over the life of the product
  • a faulty economic capital model may lead to the firm being unable to meet unexpected losses with risk capital and lead to bankruptcy

How does model risk manifest itself?

Model risk is a “second order” risk (it is linked to other financial and operational risks that are being managed using models). Therefore it can be difficult to identify the impact of model risk as a distinct risk type.

For example, an unanticipated excess of credit losses by a lending institution can be meaningfully linked to a realization of model risk only if many other related possible causes have been excluded:

  • properly developed and validated models were in place (Model Governance)
  • model results and signals were correctly and actively used as a key inputs in decision making (Model Use)


Under the assumption that all the above were in place we may conclude that the models were in fact faulty (in that they attributed systematically wrong probabilities to the possible outcomes - Intrinsic Model Risk). Hence with those caveats we can more confidently attribute any unexpected financial / reputation loss to the model infrastructure

Mitigation

As any other risk type, model risk can be mitigated in part. A prerequisite is the increased recognition, scrutiny and attention, especially given its abstract nature. A key tool for managing model risk is typically offered by a competent and independent Model Validation function.

Model risk can potentially be managed with

  • In-depth model validation which may include any of the following:
    • Self assessment: The developer is directly asked to evaluate his/her model and document the analysis
    • Documentation standards: The developer is asked to thoroughly document developments (Reproducible Research)
    • Peer review: Other developers/experts are asked to opine on a model
    • Independent Validation: Specialized and independent internal or external validation experts are asked to opine
  • Use of conditions, limits and other controls that are explicitly aligned to used models
    • Conditions for accepting a model (set in formal policy)
    • Amount of business that is relying on a new model
    • Amount of monitoring required for a new model
    • Setting limits on how long a model can be used before being re-examined, re-calibrated, re-developed etc
  • Use of model risk reserves (retain PnL)
    • Linked to the sensitivity of model outputs (e.g. valuation) to uncertain / unobservable parameters
  • Use model risk Economic Capital
    • Assessing risk capital using e.g., worst case loss analysis

Incentives of the organization to pro-actively manage model risk

Model risk is subject to risk-reward calculations where the cost of developing and validating improved models must be set against the risk posed by poorer models. Factors that may enter such an analysis:

  • Cost of funding and the degree to which model risk may be a factor in investor / market participant views of the firm
  • Regulatory view of model risk and minimum capital requirements.
  • Capitalization of model risk in internal capital assessments

Issues and Challenges

  • Modelers (quants) may have adverse incentives when developing models
  • Model users may have limited technical skills to evaluate models and may rely on third party assurances
  • Implemented models might be proprietary and opaque