Lambert found a short article by Richard Cook that I’ve embedded at the end of the post. I strongly urge you to read it in full. It discusses how complex systems are prone to catastrophic failure, how that possibility is held at bay through a combination of redundancies and ongoing vigilance, but how, due to the impractical cost of keeping all possible points of failure fully (and even identifying them all) protected, complex systems “always run in degraded mode”. Think of the human body. No one is in perfect health. At a minimum, people are growing cancers all the time, virtually all of which recede for reasons not well understood.
The article contends that failures therefore are not the result of single causes. As Clive points out:
This is really a profound observation – things rarely fail in an out-the-blue, unimaginable, catastrophic way. Very often just such as in the MIT article the fault or faults in the system are tolerated. But if they get incrementally worse, then the ad-hoc fixes become the risk (i.e. the real risk isn’t the original fault condition, but the application of the fixes). https://en.wikipedia.org/wiki/Windscale_fire#Wigner_energy documents how a problem of core instability was a snag, but the disaster was caused by what was done to try to fix it. The plant operators kept applying the fix in ever more extreme does until the bloody thing blew up.
But I wonder about the validity of one of the hidden assumptions of this article. There is a lack of agency in terms of who is responsible for the care and feeding of complex systems (the article eventually identifies “practitioners” but even then, that’s comfortably vague). The assumption is that the parties who have influence and responsibility want to preserve the system, and have incentives to do at least an adequate job of that.
There are reasons to doubt that now. Economics has promoted ways of looking at commercial entities that encourage “practitioners” to compromise on safety measures. Mainstream economics has as a core belief that economies have a propensity to equilibrium, and that equilibrium is at full employment. That assumption has served as a wide-spread justification for encouraging businesses and governments to curtail or end pro-stabilty measures like regulation as unnecessary costs.
To put it more simply, the drift of both economic and business thinking has been to optimize activity for efficiency. But highly efficient systems are fragile. Formula One cars are optimized for speed and can only run one race.
Highly efficient systems also are more likely to suffer from what Richard Bookstaber called “tight coupling.” A tightly coupled system in one in which events occur in a sequence that cannot be interrupted. A way to re-characterize a tightly coupled system is a complex system that has been in part reoptimized for efficiency, maybe by accident, maybe at a local level. That strips out some of the redundancies that serve as safeties to prevent positive feedback loops from having things spin out of control.
To use Bookstaber’s nomenclature, as opposed to this paper’s, in a tightly coupled system, measures to reduce risk directly make things worse. You need to reduce the tight coupling first.
A second way that the economic thinking has arguably increased the propensity of complex systems of all sorts to fail is by encouraging people to see themselves as atomized agents operating in markets. And that’s not just an ideology; it’s reflected in low attachment to institutions of all sorts, ranging from local communities to employers (yes, employers may insist on all sorts of extreme shows of fealty, but they are ready to throw anyone in the dust bin at a moment’s notice). The reality of weak institutional attachments and the societal inculcation of selfish viewpoints means that more and more people regard complex systems as vehicles for personal advancement. And if they see those relationships as short-term or unstable, they don’t have much reason to invest in helping to preserving the soundness of that entity. Hence the attitude called “IBY/YBG” (“I’ll Be Gone, You’ll Be Gone”) appears to be becoming more widespread.
I’ve left comments open because I’d very much enjoy getting reader reactions to this article. Thanks!