Home / Ideas Letter

Model Risk: Understanding Incorrect yet Useful Models

This article discusses an approach to understanding model risk keeping in mind that any model is a simplification and abstraction of the real world.

Ideas Letter · Vol. 7 · Aug 2010

Background

A model is a simplification of reality; it is impossible to develop a model that simulates all scenarios and produces results with 100% accuracy. Despite this, it should be acknowledged that models are insightful tools if used with judgement, skill and experience within an organization. More importantly, the capabilities of any model should be fully understood so that the model is used to make a robust decision for the right context.

Understanding Model Risk

A model is usually developed by an analyst with input from an experienced practitioner knowledgeable in a stated subject; the input usually provides a limited and constrained view of any subject matter; as is the case, in the real world, all subjects are to some degree correlated with factors that are not accounted for, and influenced by personal preferences, biases and anchors. The analyst must clearly document the assumptions and working of the model. S/he must understand the limitations of the model by back-testing the model with inputs/assumptions that produce expected output/results. The robustness of the model must be stress tested for specific inputs/assumptions; specifically, the model inputs must be changed one at a time to understand the impact of the model results. Once the results of the model (using single inputs) are clearly understood then specific stress scenarios must be tested. Finally, the model must be tested for extreme scenarios for events that are clearly outside of recent experience. It is always important to test scenarios for which expected results are available – this quantifies the variance (‘model risk’) between the expected results and real world results.

Ideas in Action

It is critical to understand the limitations of any model; it is even more important to ensure that the users of the model clearly communicate these limitations to the decision-makers who rely on the model. It should always be emphasized that models are grounded in historical experience and the decision makers should clearly understand that output/results of the world today may be different from historical experience.

Written for professional readers and reflects my views at the date shown. It is not actuarial, investment or legal advice.