bridges vol. 19, October 2008 / Pielke's Perspective

By Roger Pielke, Jr.

{enclose Vol.19_Pielke.mp3}

pielke_r_new_small.jpg "Our 21st century global economy remains regulated largely by outdated 20th century laws."  This was one of the explanations for the financial crisis given by President George Bush in his address to the nation on September 24, 2008.  While the full reasons for and details of the still-unfolding crisis will certainly be explored in depth, one important aspect of the crisis has yet to receive the attention it deserves:  the notion that regulation of the 21st century economy requires 21st century technologies in the form of highly complex financial risk models.  When the story of the current financial crisis is told in full, I expect that the misuse of risk models will be found to have played an important role.

Risk models can be valuable tools in the financial industry.  But there are two significant problems with their use in financial decision making.  One is that risk models break down in times of crisis.  Jón Daníelsson of the London School of Economics explained this dynamic in a 2000 paper appropriately titled "The Emperor has No Clothes: The Limits to Risk Modelling":  "The basic statistical properties of market data are not the same in crisis as they are during stable periods; therefore, most risk models provide very little guidance during crisis periods."  The same models that make sophisticated financial instruments possible during normal times are virtually useless during times of crisis.


{access view=guest}Access to the full article is free, but requires you to register. Registration is simple and quick – all we need is your name and a valid e-mail address. We appreciate your interest in bridges.{/access} {access view=!guest} A second problem is that the use of risk models encourages a herd mentality among firms.  According to an Inspector General's report from the US Securities and Exchange Commission released September 25, 2008, "In times of market stress, trading dries up and reliable price information is difficult to obtain.  Models therefore become relatively more important than market price in times of market stress than in times when markets are liquid and trading actively.  Such stressed circumstances force firms to rely more on models and less on markets for pricing and hedging purposes."  Daníelsson observes that the wide reliance on risk models to make decisions in a crisis can lead to perverse outcomes: "If . . . identical external regulatory risk constraints are imposed, regulatory demands may perversely lead to the amplification of the crisis by reducing liquidity."  To have many large institutions making bad decisions with flawed information is not a recipe for financial stability.

In our 2000 book on the role of geophysical predictions in decision making (Prediction, Science, Decision Making, and the Future of Nature, Island Press, 2000) we developed a set of guidelines indicating when to  rely on predictions in decision making.  The criteria are met when (1) predictive skill is known, (2) decision makers have experience in understanding and using the predictions, (3) the feedback loop between use of the prediction and evaluation of that use is relatively short (such that it can feed back into future decisions), (4) there are limited alternatives to relying on prediction, and (5) the outcomes of decisions based on predictions are highly constrained (in other words, the magnitude of the consequences of decision error is limited).

In the current financial crisis, it appears that each of these guidelines was violated: (1) decision makers have little understanding of the predictive skill of their models.  For instance, in a 2008 paper on the role of risk models in the financial crisis, LSE's Daníelsson cites a Lehmann Brothers' modeler commenting on model performance during the summer of 2007: "Events that models predicted would happen only once in 10,000 years happened every day for three days."  (2) Decision makers had little experience in using the complex risk assessments.  This was revealed dramatically during the spring of 2008, when the Financial Times reported that an error in a model used by Moody's, one of the world's most respected and widely utilized source for credit ratings, research and risk analysis, led to a far higher credit rating than was deserved by a particular complex derivative product.  Upon learning of the error, Moody's adjusted the model to reflect the ratings error, rather than admit the initial mistake.  Because no one had any experience with the sophisticated financial product being modeled, the presence of the error in the rating virtually escaped notice in the marketplace.

One could argue that the various failed institutions in the current crisis provide a good example of exactly the sort of (3) feedback that results when risk assessments are in error.  But such feedback is a lot like learning how to design and fly the space shuttle through trial and error:  Surely it can be done, but it comes at a high price.  Learning about risk models through their deployment throughout the financial system is a similarly risky practice.  (4) Effectively using models of complex, open systems usually means treating them as one of many approaches to assessing risk.  The Inspector General of the SEC has recommended that the SEC be "more skeptical" of risk models and that firms be required to develop "informal plans" for  scenarios that may not be found in their models.  In other words, they should use models heuristically and not as comprehensive tools for assessing risks.  (5) The current financial crisis will have effects that are felt for years, perhaps longer, with consequences that are not fully understood.

Risk models are an important tool and no doubt here to stay as a fundamental part of our 21st century global financial system.  But wisdom will be found in using them effectively.   As LSE's Daníelsson explains:  

The current crisis took everybody by surprise in spite of all the sophisticated models, all the stress testing, and all the numbers. The financial institutions that are surviving this crisis best are those with the best management, not those who relied on models to do the management's job. Risk models do have a valuable function in the risk management process so long as their limitations are recognized. They are useful in managing the risk in a particular trading desk, but not in capturing the risk of large divisions, not to mention the entire institution. For the supervisors the problem is even more complicated. They are concerned with systemic risk which means aggregating risk across the financial system. Relying on statistical models to produce such risk assessments is folly. We can get the numbers, but the numbers have no meaning.

Using risk models effectively in the 21st global financial system will require the widespread use of a decidedly pre-21st century tool - common sense.


Roger Pielke, Jr. is the former director of the Center for Science and Technology Policy Research (2001-2007). He has been on the faculty of the University of Colorado since 2001 and is a professor in the Environmental Studies Program and a fellow of the Cooperative Institute for Research in the Environmental Sciences (CIRES).