As we’ve discussed previously, the creaky code base in large banks is a systemic risk. Major banks run their transactions on mainframes, and significant portions of the software is both ancient and customized. Since at least the mid 1990s, these banks have had major projects to migrate off their legacy code. Although it is hard to prove a negative, I am highly confident no one has succeeded. Why? It would be such a spectacular accomplishment (and would still be very costly) that any bank that had succeeded would have broadcast its accomplishment.
A recent article in Medium, Interviewing my mother, a mainframe COBOL programmer, gives a sense of why the problem is so intractable. I’ve excerpted key sections and encourage you to read it in full (the other parts of the article have detail on database sizes and organization which experts will find informative). The mother works at a bank now known as Nordea, which has about $700 billion in assets. Contrast that with JP Morgan, which has roughly $2.4 trillion in assets, and massive derivatives clearing operation on top of that.
From the article:
This position is the most important one in the bank, at least from a technical standpoint. If, let’s say, my mother and everyone on her team would quit their job, the bank would go under within a matter of weeks if they’re lucky. They have a rotation of people on her team being available 24/7. I remember when I was younger and she had to take a taxi to work in the middle of the night on a Sunday to fix a dead-lock problem.
…is not a fancy programming language like your functional Haskell or concurrent Golang— it’s an imperative, procedural language and since 2002, object-oriented. There’s nothing wrong with the language itself, the problem is that barely anyone knows it — at least not in the context of mainframe programming. My mother is the next youngest person on her team, and she’s born 1964, and the youngest person being 2 years younger. Since almost all of the largest banks in the world runs on IBM Mainframe with COBOL as the primary programming language, this is a global issue. The smaller banks however are better off which usually runs something like Java without mainframes….
Banking systems are also extremely advanced. A personal bank account differs a lot from a business bank account, and there are at least 50 different types of bank accounts for each of them. And in Nordeas case, they also have the Swedish government accounts, which are different from both personal and business accounts. I think they have the Finnish government accounts and maybe a portion of Denmarks as well, which differs too.
Clive provided more corroborating information via e-mail:
Did you read what the most highlighted section in the piece had become:
There are programs that are decades old that nobody even knows what they do and the person who wrote it is long gone.
At my TBTF, I heard only last week there’s a routine in COBOL which is something to do with payments. It was written in the late ’60s. What it does is fairly easily discernible from the code, but what is a complete unknown is what upstream or downstream dependencies there might be for this module. It may be being invoked directly. Or its output may be being checked. Or it could have become an elaborate subroutine for another process. No-one really knows. So there it stays, untouched, undecommissionable.
If it was a lower priority system it might be possible to experiment a bit and remove it, see what happens. But if it brought down payments or, worse, didn’t bring down payment but corrupted the processing of payments in some subtle but damaging way, the losses could run into hundreds of millions.
If there was a lot more money available, we could suspend the module in dev and see what happens. But We’ve only got three or four “proper” feature-complete dev regions and these are all permanently tied up with other, urgent, projects/changes/fixes. Therefore you can’t tie them up for weeks (or more likely months) doing month-end, quarter-end and year-end simulations. We’ve got a fair few more other development systems, but these are not complete with all the interfaces, so are only suitable for unit testing which means you can’t do the sort of end-to-end testing needed to really get under the hood of this routine.
Simplest, then, to leave it sitting there. Like one of the monoliths in 2001, a beguiling mystery.
A couple of years ago, there was a big programme started to decommission a load of “legacy” systems. No-one can think of any system which has actually been decommissioned.
At the C-suite level, the drive for bodyshopification continues unrelentingly — take experienced, knowledgeable but expensive subject matter experts, shovel them out and get in someone (maybe two or even three different people, it doesn’t seem to matter) at 1/3rd the cost in India, the Philippines, Poland, it really doesn’t bother anyone at all where, instead. All they need to do is have a months’ handover and get a link to a SharePoint site with the “documentation” and off they go. The churn is immense, these “resources” are rarely there for more than a year, and never there for more than two. Oh, and every so often, the whole outsourcer gets the heave-ho to be replaced by another outsourcer who can cut the day rate by a few pounds here or there.
No, it doesn’t feel like it will end well. But the plates keep spinning in the air for now, so nothing is going to change in the meantime…
On one of our earlier discussions of bank IT risk, a knowledgeable reader said (and I wish I could locate the comment) that the reason banks haven’t migrated off the legacy code is that the idea stops dead when management realizes that it will take three years of their profits to get it done. And that’s before you get to the fact that massive cost overruns in IT projects are routine, and 80% of large IT projects fail.
It’s inconceivable that regulators are not aware of this ticking time bomb, and yet are not forcing banks to act. When this all blows up is anyone’s guess, but a train wreck is inevitable, if nothing else due to the members of the mission-critical teams that baby this ancient code retiring and dying.