Comments on naked capitalism: "Blame the Models"

Amen. Or Hallelujah?It's saying what I've been say...

2008-05-09T07:02:00.000-04:00

Amen. Or Hallelujah?
It's saying what I've been saying for a few years now - we can generate very precise but also highly innaccurate number and we fall in love with the precision and say it's accuracy (Disclaimer: I do models myself. Often. And I don't trust them more than my intuition.).

Yves:"For reasons I cannot fathom (perhaps the rise of the PC and the ease of slicing and dicing data), qualitative assessments are seen as inferior to quantitative ones."
It's easy. Management is hard, models are easy (there are no 'soft skills' involved). Making decisions based on experience and intuition (and getting it right more often than not) is harder than having set, inflexible, rules that you're supposed to follow. Remember, by definition most of the people you will hire are average or below average (if you're over certain size, anyways. Getting another good person is harder than the previous one because the universe of really good people is small and there's a lot of competition for them), and people who are really good (as opposed who got lucky a few times - which you might not be even able to tell for a while) are rarer still.

And Francois/Richard K are spot on with the blame game.

Models cover the boss's ass while judgments expose...

2008-05-09T03:50:00.000-04:00

Models cover the boss's ass while judgments expose it; this is their greatest utility. I'm not half joking when I say that. (Ohh _don't_ let me get started on models---and I build them all the time in my areas of expertise!) The reason why the use of models has grown is that top management in most organizations are corporate hacks [I misspelled that, but let's be polite] looking to make their bonuses playing with the customers' money rather than speculator-entrepreneurs betting their own. John D. Rockefeller, nasty ole J. P. Morgan, George Heast, Charles Merrill, George Soros, Warren Buffet: what's the common factor? Yes, that's right. We can be sure that all of them read the numbers, but then they put their 'expert systemic judgment' to work before they made their play. I'll trust the expert judgment on the back of an envelope of an experience trader who has won and lost bets over time over any model ever made. The trader will be wrong at times but they'll be wrong for a reason: the model will be wrong for no reason. And that's just . . . wrong.

Lots of comments on this one, not enough time.Let'...

2008-05-09T03:45:00.000-04:00

Lots of comments on this one, not enough time.

Let's take a biggie, the chance of a rogue-trading loss. I kid you not, the way Basel II was implemented at my bank (and, as near as I can tell, at others) was (roughly) to calculate the amount of rogue-trading losses and divide by the universe of banks of which we considered ourselves a part of. Then (of course) we adjusted downwards because we considered our risk management superior... And voila, we knew the probability of the loss, and how much capital we were supposed to put aside.

"They know that statistical models have major shortcomings, particularly in underestimating the odds of catastrophic losses, which is precisely what they are supposed to help avoid."

I'd say that this is just not true. IBs don't care about catastrophic losses, they care about large day-to-day losses. The latter are more frequent and result in the same outcome: no bonus and/or firing.

Could it be that models are convenient refuges pre...

2008-05-09T01:31:00.000-04:00

Could it be that models are convenient refuges precisely in a time of crisis?

It is easier to claim that "Our models have failed." than to have to admit "We were wrong".

Also, I wonder if anchoring could explain in part, the paradox Yves alluded to ie. How can we simultaneously mistrust models and advocate their use?

Reducing a complex phenomenon to a set of numbers can be seductive; like doctors who have a tendency to treat lab numbers instead of the patient they have in front of them.

"Therefore one of the most important lessons from ...

2008-05-08T18:18:00.000-04:00

"Therefore one of the most important lessons from the crisis has been the exposure of the unreliability of models and the importance of management."

Yes, but will it be learned? The point you don't make is that finance is driven by these models. It is questionable how it would be controlled without them. I'm not saying the control is good but that fooling with it could bring on some serious instabilities.

If government would regulate the OTC instruments r...

2008-05-08T18:14:00.000-04:00

If government would regulate the OTC instruments responsible for the tight coupling of the ibanks, no one would need to care if risk managers use 5000 dimension multi-variate models or crayons and kraft paper.

Great post, Yves. From my experience, these though...

2008-05-08T18:13:00.000-04:00

Great post, Yves. From my experience, these thoughts are directly on the mark, and explain most of why we saw the recent banking behavior in mortgage markets.

Danielsson's point goes well beyond SIVs, however. He seems to have a great deal of faith in the internal risk modeling taking place in individual banks. Having worked in risk management at a couple of large banks, I'm not as confident in those models, primarily because of the "garbage-in, garbage-out" syndrome. All of the models require inputs, and those inputs are very often more subjective than objective, and thus quite subject to interference from management at all levels, which is usually unwilling to allow inputs to reflect negatively on their own team's performance.

Your post also brings up another fascinating possibility: does the uncertainty principle in physics apply to complex information systems like financial markets? The famous experiments demonstrating that principle suggests that light quanta "know" when a measurement is being taken.

Is it much of stretch to be able to demonstrate that sentient actors in the financial markets are "quanta" that "know' when a measurement is being taken, and change their behavior accordingly? Isn't that part of the underlying reason why observed behaviors such as the "January effect" in stock can't be consistently exploited over time?

Philosophically, the fundamental issue seems to be that management seems to have generally lost confidence in it's own ability to judge risk in financial markets, and instead is religiously relying on objective, quantifiable data elements.

While objective data is undoubtedly a critical input in the decisionmaking process, the point seems to be that some level of human judgment is required, if only in order to allow for the fact that no model can contain all possible relevant factors.

Clearly, you have always taken the drawbacks (and ...

2008-05-08T17:38:00.000-04:00

Clearly, you have always taken the drawbacks (and underlying assumptions) of the model into account. No quant will deny that (and every model has these).

I want to add here another problem: People are often not allowed to use models with thinking. I give you a simple example: last december I had an interview for a quant analyst position for a big and well-known investment company. They used a (roughly) extended CAPM model for the asset allocation. However, they feed the model mostly with complete old data - ignoring completely the underlying fundamentals. I argued that equities must habe negative drift in the next months. Everything else doesn't make sense at all. They said its bullshit and threw me out of the office (so I did not got the job). No joke!!