Bank IT, Grexit, and Systemic Risk

The fact that Greece is submitting to being shoved hard into even deeper economic misery and the loss of much of its sovereignity should in and of itself serve as an indirect proof of the fact that a Grexit would be even worse. We’ve argued that the acute distress created by two weeks of a bank holiday was only a mild foretaste of what was in store. Greece would lose much if not all of its tourist industry (18% of GD) and would find it difficult to impossible to import, when Greece is not self sufficient in food and imports petroleum and pharmaceuticals. Experts were warning that if the bank holiday persisted much longer, not only would more businesses fail, but food shortages, which were starting to take hold, would become a problem by the end of July.

Readers had a great deal of difficulty accepting that in the case of a Grexit, it would take far longer than they imagined to get physical currency in circulation, and that that was actually easy compared to the systems issues. Many commentors seemed unable to grasp how they are so dependent on functioning payment systems that they have difficulty thinking through what not having them implies. Try imaging a world without an electrical grid for the foreseeable future. While the decay slower than that of a power outgage, the level of disruption over time becomes similar.

Most experts contend that it would take a minimum of six months, even with advanced planning, to get drachma printed and distributed; typical estimates are 12 to 18 months. Yanis Varoufakis gave a simple illustration:

In occupied Iraq, the introduction of new paper money took almost a year, 20 or so Boeing 747s, the mobilisation of the US military’s might, three printing firms and hundreds of trucks. In the absence of such support, Grexit would be the equivalent of announcing a large devaluation more than 18 months in advance: a recipe for liquidating all Greek capital stock and transferring it abroad by any means available.

On the IT front, the challenge is vastly larger due to the state of financial firm IT systems. We intend to return to this topic, because we see bank IT systems as an unrecognized source of systemic risk. They are required to run to mission critical standards: enormous transaction volumes, extremely high demands for accuracy of end output, high uptimes. Yet the code base is an aggolmeration, with many important operations relying in meaningful ways on legacy systems. Thus, as our expert with relevant experience stressed, changes that seem simple are anything but.

Hoisted from comments on the post Convert to the Drachma – Piece of Cake. Right…. First from the post’s author, Louis Proyect:

There is nothing difficult about it at all. The issue is finding program code in billions of lines of code to make the proper change. After I left Goldman, I finally ended up at Columbia University, which had a pretty good IT infrastructure for a nonprofit. We spent a year analyzing Y2K conversion tasks and then a year testing it. And all that was involved for the most part was looking for program code that was in an mm/dd/yy format and modifying to mm/dd/yyyy.

And Troutwaxer elaborated:

The scope of the problem is more like this. Imagine that you need to make a change to the canon of Western Literature, such that your brand-new metaphor makes sense to everyone who needs it. The “source code” for Western Literature goes back to ancient Greece, (and maybe even Sumeria) such that Shakespeare was making references to documents written thousands of years before his time, originally in ancient versions of Greek or Latin.

In order to make your “small change” to the canon of Western Literature, you need to rewrite every book, every scroll, every stone slab ever created to contain pointers to the metaphor you wish to create, and you need to do it perfectly and transparently. You need the cooperation of every literary figure from Homer to Bob Dylan, and of every publisher and librarian from Random House to the elementary school on the corner.

Is it doable? Of course, but we’re talking millions of person-hours.

Back in the real world, the one, single tiny problem of “under which circumstances your new code should produce the Drachma symbol as opposed to the Euro symbol” is gigantic, and probably requires hundreds of hours of meetings merely to produce a requirements document.

To make matters worse, I’m guessing that the Eurocrats will do everything possible to make sure the source code is not available.

The only possibility I can see is for the Greeks to start an open-source program to build a banking system, and they’re six months too late. (Actually, more like six years too late, but Tsipras wasn’t there six years ago.)

Clive stressed not just that critical code was old, but that management lacks a grasp of how key elements of the machinery work:

The back end legacy systems in virtually all of the TBTFs are running — in some parts, not all certainly, but there are some modules — stuff that is 30+ years old…

You’d be shocked at just what goes on in the data centres of some of the largest, most critical banks. Never, for one single second, assume that it is either a) even vaguely modern or, perhaps more worryingly b) there is anyone in the owning organisation who really knows how it actually works.

The Greek banks could hardly have been drowning in systems investment budget in the past 5 to 10 years. I shudder to think what they are like under the hood.

And Clive also disabused a reader of the notion that the systems were modularlized or otherwise confirmed to what were recognized as good programming practices. I saw this in operation back at O’Connor, which was then seen as the most cutting edge IT shop in financial services, and arguably the private sector (it ran the largest Unix network in the world; I’d see top IT firms come in and meet with the O’Connor IT people. The visitors’ eyes would get as big as saucers when the O’Connor folk gave them an idea of what they were up to. They were also fiendish about protecting their intellectual property; only a very few people had access cards that would allow them to go back into the parts of the firm where the programmers worked, and even those areas were physically Balkanized). Even at a firm that recognize technology as key to its competitive and spent 50% of its expenses on it, the CIO could not get his partners to pay for documenting the programming, which would have been cheaper over the development cycle, because it would have increased up front time and costs by roughly 20%.

Clive disabused a reader of the notion that the fact that application and presentation levels stuff wasn’t modularized meant the IT staff should be sacked:

When I read comments along the lines of this one, to summarise the argument being made which is “why oh why isn’t it the case that (insert modern software development practice here) is not done in (in this case, your typical TBTF’s core systems) they’d be mad / incompetent / lazy not to do so” I’m reminded of stale arguments which have been made ad nauseam at these institutions and have become well settled to the point where no-one there really bothers any more with them.

I’ll issue a reminder here which should be superfluous but perhaps isn’t. These are businesses. They are in the business of making money. They are ruthlessly focussed at this and c-level execs live and die, career-wise, by the quarterly results. Controlling controllable costs is the second highest priority, right behind finding new and inventive ways to extract profits from customers. Any c-level direct report who, and they would most assuredly be seen in this way, waffles on about technical purity would get a polite hearing once or twice, but if they persisted that would be the end of them because they’d be committing that cardinal sin in a TBTF’s culture of not being sufficiently business minded.

While what you say about the benefits of modular, re-usable and even platform independent code is true, in the maybe 5 minutes you’d have to make your pitch for it to the c-level exec who holds the investment purse strings (they have attention spans that would shame a mayfly, usually) you’d better be able to say, precisely, how much cost rewriting and existing spaghetti code would save and what it would take to get to the end result.

Every single penny which is reluctantly allocated for IT investment at a TBTF (and similar enterprises, the way the landscape is shaped is very consistent across the industry) has to satisfy a 5-year (typically) business case. It must not only make up the costs of the project, it must pay back within that timeframe and show a return — 20% is the minimum.

Oh, and I should also mention that TBTF-scale enterprises are riven with politicking and factionalism at every turn. Squabbles and backstabbing abound and are usually around who is getting the funding, how much and how important they appear to be to the CEO. Gifting influence and power to the IT department via generous funding increases the IT department’s profile to the inevitable detriment of others. So naturally any funding request for IT spend has to be set against that power dynamic.

Of course, you can put non financial benefits in a business case. But the way business case construction works means that it is very bad at capturing non financial benefits properly. NC has covered in depth short-termism in modern capitalism at length in previous features. The only way that non financial benefits get a look-in is if the company is facing clear, onerous, immediate risks. To give an example, one of the key systems at a TBTF was running on a mixture of COBOL and assembler and was a nightmare to maintain. But it wasn’t until it got to the point where programmers with good, solid assembler skills become almost impossible to find — at any price — that the risk mitigation of porting the application to vendor neutral C got green-lighted.

So it is not at all right to say that developers are incompetent. Any bullets fired in their direction are misaimed. But — and the parallels of a recently departed finance minister strike me here — you can’t be a technical purist hold-out alone, you have to appreciate the investment climate, any political machinations in play and the fact that producers (of earnings) are given about 10x the credibility and bandwidth that cost centres are.

The fact that the producers treat bank IT as a costly afterthought and those professionals been doing development in a workable-at-best, as opposed to workmanlike manner for decades, means that codebase is getting creakier and creakier. Wall Street IT greybeards will tell you that even as of 20 years ago, projects to replace legacy code pretty much never get done. Now that the systems are even bigger and more complex due to increasing product and geographic scope as well as mergers, means that the legacy code issue and the ever-increasing complexity of the systems is an accident waiting to happen. We plan to return to this topic, so stay tuned.

Print Friendly, PDF & Email