Recent Items

"Blame the Models"

Posted on by

One of our pet interests has been how the use of mathematics and models can unwittingly enable people to fool themselves. We see this regularly when working on deals. The model for the target business’ performance somehow becomes more real than the company. When the numbers don’t work, if you can come up with a good sounding rationale for tweaking them, presto! Suddenly everything in hunky dory. No wonder over 60% (some studies say as high as 75%) of all deals fail.

Our colleague Susan Webber, in an article about the corporate obsession with metrics, made some pertinent observations:

Metrics presuppose that situations are orderly, predictable, and rational. When that tenet collides with situations that are chaotic, nonlinear, and subject to the force of personalities, that faith—the belief in the sanctity of numbers—often trumps seemingly irrefutable facts. At that point, the addiction begins to have real-world consequences. Business managers must recognize the limitations of metrics.

Mind you, I’m not arguing that metrics and measurement are inherently bad things. To note just one example, a well-structured performance measurement system is essential to the well-being of large enterprises. But quantitative measures can be and frequently are used naively. It’s all too easy to abdicate judgment to the output of a model or scorecard.

Jon Danielsson at VoxEU takes this viewpoint further in an article that discusses a pervasive cognitive dissonance among trading operations and their regulators. They know that statistical models have major shortcomings, particularly in underestimating the odds of catastrophic losses, which is precisely what they are supposed to help avoid. While the conventional response has been to try to devise better models, Danielsson argues that that line of thinking is wrongheaded.

For Danielsson makes a fundamental point: what matters is management; the models are secondary. For reasons I cannot fathom (perhaps the rise of the PC and the ease of slicing and dicing data), qualitative assessments are seen as inferior to quantitative ones. But for a regulator to understand the robustness of a company’s management practices requires more scrutiny than has been fashionable of late. And it also requires better regulators.

From VoxEU:

In response to financial turmoil, supervisors are demanding more risk calculations. But model-driven mispricing produced the crisis, and risk models don’t perform during crisis conditions. The belief that a really complicated statistical model must be right is merely foolish sophistication.

A well-known American economist, drafted during World War II to work in the US Army meteorological service in England, got a phone call from a general in May 1944 asking for the weather forecast for Normandy in early June. The economist replied that it was impossible to forecast weather that far into the future. The general wholeheartedly agreed but nevertheless needed the number now for planning purposes.

Similar logic lies at the heart of the current crisis

Statistical modelling increasingly drives decision-making in the financial system while at the same time significant questions remain about model reliability and whether market participants trust these models. If we ask practitioners, regulators, or academics what they think of the quality of the statistical models underpinning pricing and risk analysis, their response is frequently negative. At the same time, many of these same individuals have no qualms about an ever-increasing use of models, not only for internal risk control but especially for the assessment of systemic risk and therefore the regulation of financial institutions.1 To have numbers seems to be more important than whether the numbers are reliable. This is a paradox. How can we simultaneously mistrust models and advocate their use?…..

Underpinning this whole process is a view that sophistication implies quality: a really complicated statistical model must be right. That might be true if the laws of physics were akin to the statistical laws of finance. However finance is not physics, it is more complex, see e.g. Danielsson (2002).

In physics the phenomena being measured does not generally change with measurement. In the finance that is not true. Financial modelling changes the statistical laws governing the financial system in real-time. The reason is that market participants react to measurements and therefore change the underlying statistical processes. The modellers are always playing catch-up with each other. This becomes especially pronounced when the financial system gets into a crisis.
This is a phenomena we call endogenous risk, which emphasises the importance of interactions between institutions in determining market outcomes. Day-to-day, when everything is calm, we can ignore endogenous risk. In crisis, we cannot. And that is when the models fail.

This does not mean that models are without merits. On the contrary, they have a valuable use in the internal risk management processes of financial institutions, where the focus is on relatively frequent small events. The reliability of models designed for such purposes is readily assessed by a technique called backtesting, which is fundamental to the risk management process and is a key component in the Basel Accords.

Most models used to assess the probability of small frequent events can also be used to forecast the probability of large infrequent events. However, such extrapolation is inappropriate. Not only are the models calibrated and tested with particular events in mind, but it is impossible to tailor model quality to large infrequent events nor to assess the quality of such forecasts.

Taken to the extreme, I have seen banks required to calculate the risk of annual losses once every thousand years, the so-called 99.9% annual losses. However, the fact that we can get such numbers does not mean the numbers mean anything. The problem is that we cannot backtest at such extreme frequencies. Similar arguments apply to many other calculations such as expected shortfall or tail value-at-risk. Fundamental to the scientific process is verification, in our case backtesting. Neither the 99.9% models, nor most tail value-at-risk models can be backtested and therefore cannot be considered scientific.

We do however see increasing demands from supervisors for exactly the calculation of such numbers as a response to the crisis. Of course the underlying motivation is the worthwhile goal of trying to quantify financial stability and systemic risk. However, exploiting the banks’ internal models for this purpose is not the right way to do it. The internal models were not designed with this in mind and to do this calculation is a drain on the banks’ risk management resources. It is the lazy way out. If we don’t understand how the system works, generating numbers may give us comfort. But the numbers do not imply understanding.

Indeed, the current crisis took everybody by surprise in spite of all the sophisticated models, all the stress testing, and all the numbers. I think the primary lesson from the crisis is that the financial institutions that had a good handle on liquidity risk management came out best. It was management and internal processes that mattered – not model quality. Indeed, the problem created by the conduits cannot be solved by models, but the problem could have been prevented by better management and especially better regulations.

With these facts increasingly understood, it is incomprehensible to me why supervisors are increasingly advocating the use of models in assessing the risk of individual institutions and financial stability. If model-driven mispricing enabled the crisis to happen, what makes us believe that the future models will be any better?

Therefore one of the most important lessons from the crisis has been the exposure of the unreliability of models and the importance of management. The view frequently expressed by supervisors that the solution to a problem like the subprime crisis is Basel II is not really true. The reason is that Basel II is based on modelling. What is missing is for the supervisors and the central banks to understand the products being traded in the markets and have an idea of the magnitude, potential for systemic risk, and interactions between institutions and endogenous risk, coupled with a willingness to act when necessary. In this crisis the key problem lies with bank supervision and central banking, as well as the banks themselves.

Print Friendly, PDF & Email

8 comments

  1. Anonymous

    Clearly, you have always taken the drawbacks (and underlying assumptions) of the model into account. No quant will deny that (and every model has these).

    I want to add here another problem: People are often not allowed to use models with thinking. I give you a simple example: last december I had an interview for a quant analyst position for a big and well-known investment company. They used a (roughly) extended CAPM model for the asset allocation. However, they feed the model mostly with complete old data – ignoring completely the underlying fundamentals. I argued that equities must habe negative drift in the next months. Everything else doesn’t make sense at all. They said its bullshit and threw me out of the office (so I did not got the job). No joke!!

  2. Anonymous

    Great post, Yves. From my experience, these thoughts are directly on the mark, and explain most of why we saw the recent banking behavior in mortgage markets.

    Danielsson’s point goes well beyond SIVs, however. He seems to have a great deal of faith in the internal risk modeling taking place in individual banks. Having worked in risk management at a couple of large banks, I’m not as confident in those models, primarily because of the “garbage-in, garbage-out” syndrome. All of the models require inputs, and those inputs are very often more subjective than objective, and thus quite subject to interference from management at all levels, which is usually unwilling to allow inputs to reflect negatively on their own team’s performance.

    Your post also brings up another fascinating possibility: does the uncertainty principle in physics apply to complex information systems like financial markets? The famous experiments demonstrating that principle suggests that light quanta “know” when a measurement is being taken.

    Is it much of stretch to be able to demonstrate that sentient actors in the financial markets are “quanta” that “know’ when a measurement is being taken, and change their behavior accordingly? Isn’t that part of the underlying reason why observed behaviors such as the “January effect” in stock can’t be consistently exploited over time?

    Philosophically, the fundamental issue seems to be that management seems to have generally lost confidence in it’s own ability to judge risk in financial markets, and instead is religiously relying on objective, quantifiable data elements.

    While objective data is undoubtedly a critical input in the decisionmaking process, the point seems to be that some level of human judgment is required, if only in order to allow for the fact that no model can contain all possible relevant factors.

  3. Steve

    If government would regulate the OTC instruments responsible for the tight coupling of the ibanks, no one would need to care if risk managers use 5000 dimension multi-variate models or crayons and kraft paper.

  4. Anonymous

    “Therefore one of the most important lessons from the crisis has been the exposure of the unreliability of models and the importance of management.”

    Yes, but will it be learned? The point you don’t make is that finance is driven by these models. It is questionable how it would be controlled without them. I’m not saying the control is good but that fooling with it could bring on some serious instabilities.

  5. Francois

    Could it be that models are convenient refuges precisely in a time of crisis?

    It is easier to claim that “Our models have failed.” than to have to admit “We were wrong”.

    Also, I wonder if anchoring could explain in part, the paradox Yves alluded to ie. How can we simultaneously mistrust models and advocate their use?

    Reducing a complex phenomenon to a set of numbers can be seductive; like doctors who have a tendency to treat lab numbers instead of the patient they have in front of them.

  6. a

    Lots of comments on this one, not enough time.

    Let’s take a biggie, the chance of a rogue-trading loss. I kid you not, the way Basel II was implemented at my bank (and, as near as I can tell, at others) was (roughly) to calculate the amount of rogue-trading losses and divide by the universe of banks of which we considered ourselves a part of. Then (of course) we adjusted downwards because we considered our risk management superior… And voila, we knew the probability of the loss, and how much capital we were supposed to put aside.

    “They know that statistical models have major shortcomings, particularly in underestimating the odds of catastrophic losses, which is precisely what they are supposed to help avoid.”

    I’d say that this is just not true. IBs don’t care about catastrophic losses, they care about large day-to-day losses. The latter are more frequent and result in the same outcome: no bonus and/or firing.

  7. Richard Kline

    Models cover the boss’s ass while judgments expose it; this is their greatest utility. I’m not half joking when I say that. (Ohh _don’t_ let me get started on models—and I build them all the time in my areas of expertise!) The reason why the use of models has grown is that top management in most organizations are corporate hacks [I misspelled that, but let’s be polite] looking to make their bonuses playing with the customers’ money rather than speculator-entrepreneurs betting their own. John D. Rockefeller, nasty ole J. P. Morgan, George Heast, Charles Merrill, George Soros, Warren Buffet: what’s the common factor? Yes, that’s right. We can be sure that all of them read the numbers, but then they put their ‘expert systemic judgment’ to work before they made their play. I’ll trust the expert judgment on the back of an envelope of an experience trader who has won and lost bets over time over any model ever made. The trader will be wrong at times but they’ll be wrong for a reason: the model will be wrong for no reason. And that’s just . . . wrong.

  8. vlade

    Amen. Or Hallelujah?
    It’s saying what I’ve been saying for a few years now – we can generate very precise but also highly innaccurate number and we fall in love with the precision and say it’s accuracy (Disclaimer: I do models myself. Often. And I don’t trust them more than my intuition.).

    Yves:”For reasons I cannot fathom (perhaps the rise of the PC and the ease of slicing and dicing data), qualitative assessments are seen as inferior to quantitative ones.”
    It’s easy. Management is hard, models are easy (there are no ‘soft skills’ involved). Making decisions based on experience and intuition (and getting it right more often than not) is harder than having set, inflexible, rules that you’re supposed to follow. Remember, by definition most of the people you will hire are average or below average (if you’re over certain size, anyways. Getting another good person is harder than the previous one because the universe of really good people is small and there’s a lot of competition for them), and people who are really good (as opposed who got lucky a few times – which you might not be even able to tell for a while) are rarer still.

    And Francois/Richard K are spot on with the blame game.

Comments are closed.