As we’ve written about how the complexity of payment systems and the creaky legacy which sits at the core of the transaction processing engines of banks, which perversely run fault intolerant, mission critical operations on such a shaky foundation, the more astute readers, once they get over their incredulity, quickly recognize that this combination isn’t just a Grexit obstacle, but also a source of systemic risk.
Our Richard Smith, who has extensive experience in capital markets IT, first mentioned this looming problem years ago, but it did not seem timely to make it a major focus. But we are seeing more and more evidence that legacy systems are starting to break down at major firms, both from media stories and some private accounts we’ve received.
Members of our audience who have technology experience but have not dealt with either large lumbering corporate and/or major financial firm information technology infrastructure and assert that we are exaggerating how bad things are have revealed that they don’t know what they don’t know. Some comments from recent posts:
What many of you “it’s easy” people fail to understand is that mainframe programming is nothing like today’s coding. COBOL, PL/I etc. do not support modern concepts like objects, polymorphism or anything else. Think assembly language with nicer mnemonics. XML ? Hah, there is virtually no such thing for the mainframe. There’s no git, no mercurial etc. Virtually none of the tools that exist for Wintel/Linux are available to mainframers.
In large organizations there are hugely cumbersome change management processes. Where I am, a simple code change might take a minimum of eight weeks to deploy, and we only have a dozen systems. Actual application changes like envisioned here would take at least six to twelve months for coding and testing, and then another four months for deployment. For large banks, I would expect the timeframes to be even longer because the systems are so critical.
Anyone who says it’s trivial simply has zero knowledge of the large systems environment.
All modern systems become a part of the legacy patchwork, because no enterprise ever completely replaces all its legacy systems. My experience is little gets discarded and the integration links to other system multiply over time. Think n factorial in linkages or interfaces, where n increases every year.
I’m in the IT biz, h/w not s/w but still …, Louis & Clive (+ other commentators) have pretty much nailed the difficulties of modifying, testing, and then deploying new code into an exiting, running, mission-critical system stuffed fully of dusty deck legacy. They are legion even before you get to what might be called the “user interface” aspects – in this case POSs, ATMs, PIN delivery, check clearing etc. Its not called spaghetti code for nothing (*).
However what has struck me is that there’s an assumption here that seems to be going without question: All the s/w that has to be changed is still available in source code form – be it COBOL, PL/1, Assembler, Fortran or whatever – and can be recompiled. Maybe in amongst the decades old stuff are binary blobs of machine code whose source has long since been lost and whose interface is barely defined/understood, if at all.
One thing that a lot of people who aren’t dealing with software development day-to-day don’t realise is that while writing good code is hard enough, reverse engineering it is an order of magnitude or two harder and that’s when you’re dealing with good code.
For the old banking systems, you have to figure out the intent of the original author 30-40 years back, plus an accumulated, monstrous hairball of quick fixes. At this point it doesn’t matter if the code is OO, functional, procedural or handcrafted assembly written by someone stoned out of their minds. The programmers working to maintain the system normally aren’t the best, brightest and most senior either, more likely the cheapest and just-about-adequate ones because maintenance isn’t sexy and whatever top notch programmers the financial institution has are working on new projects that also have deadlines attached to them and often tight budgets.
There aren’t a lot of competent people that are willing and able to wander into that morass and do what it takes to get it cleaned up and fix the issue. In fact you may find more willing people than ones that are able – I made a pretty good living for a decade dealing with first generation trading systems (ie, stuff developed in the late 80s/early 90s and then carried forward for decades) that needed fixing and updating. Original authors are long gone, documentation is, err, sparse and you have a tight deadline because as usual in investment banking, everything needs to be completed yesterday.
Now look at an interconnected payment system that is even older, created using languages and technologies that people have been actively avoiding for decades and see how many people you can find who can do even a half baked job.
Yves here. Let’s look at an example of a major bank whose IT systems have started to fall apart, RBS. We’ve taken sections from a detailed July Herald Scotland article, The ticking time-bomb at the heart of our big banks’ computer systems. I strongly urge you to read it in full. The article begins by describing how a half million transactions simply disappeared from RBS’ systems in June. Here are some of the high points:
Experts have for years been warning that the legacy systems of high street banks – some dating back to the 1960s – are an IT disaster waiting to happen. Systems built on costly and unwieldy platforms regularly buckle under the strain amid warnings over vulnerability to hacking attacks.
Last year the 79 per cent taxpayer-owned RBS paid a record £56 million fine following a much larger systems outage in 2012 which left millions alienated from their cash. In 2013 a further glitch left customers unable to pay by card or withdraw cash on one of the busiest shopping days of the years….
Although RBS has the worst record IT failures, there have also been well-publicised outages at Lloyds, the Cooperative Bank, TSB and the Bank of Scotland.
And the UK’s banks are not alone. The technological meltdown narrative is mirrored worldwide in the sector. Last year banks in Europe spent an estimated £40 billion on IT but only £7bn of that was investment in new systems: the remaining £33bn was spent patching and maintaining increasingly creaky and fragmented legacy systems.
The big players throughout the developed world (with the exception of countries such as Turkey and Poland whose banks were slower to install computers in the sixties and seventies and, as a result, have fewer legacy issues to cope with now) use computer systems that have been built up and adapted over several decades.
Acquisitions have led to compromises being found and new product launches have led to bolt-on systems being patched onto layers of other complex systems surrounding the “legacy core” – typically a 1960s mainframe which carries out the bank’s central ledger functions.
A report by Deloitte from as far back as 2008 said that “many banks have now reached a tipping point where the cost and risk of doing nothing outweighs the cost and risk of taking action”. And yet, seven years on, little has since changed.
According to Chris Skinner, author of a leading book on digital banking and chairman of the Financial Services Club, banks don’t want to take the risk of replacing their “groaning” legacy systems because any serious outage would lead to more fines from regulators and could even put them out of business. Because the stakes are so high, there has been a lack of leadership in the industry to tackle the issue.
There is a kinda-sorta solution as our Richard Smith points out:
Rather than taking the plunge and replacing entire systems there is now a move among some multinational operators to set up new smaller banks unencumbered by legacy issues. A notable example was the successful launch in 2013 by BNP Paribas of its new Hello Bank! in four European countries.
While most banks keep their hardware up to date the opposite is true of their software according to veteran capital markets analyst and blogger Richard Smith.
“Software in the industry can be very ancient indeed, and if the software is very old it doesn’t really matter how new the hardware is,” he says.
Many of the designers of the original bank software are now retired or dead and this loss of knowledge about their own core systems has been exacerbated by outsourcing data operations overseas, principally India, to reduce costs.
“Cheap inexperienced graduates were employed overseas and knowledge was lost in the UK,” observes Smith. “The banks decided to go for cost reduction rather than reliability and maintainability and they now have an accumulating and accelerating problem brought on by the loss of that knowledge.”
“The problem was evident to the far-sighted 15 to 20 years ago but it has now become a runaway problem with an awful lot of momentum and it’s getting worse.”
Why is this of setting up entirely new banks less germane than it appears? The most immediate problems are showing up at retail banks, and it’s also those types of businesses and services that are easier to balkanize and move onto other platforms (and “easier” is not “easy”). But where a seize-up would have the most serious ramifications is the wholesale banking and capital markets operations of the major international banks. And we’ve heard reports of 24/7 emergency operations to keep those systems running after some serious processing issues at a major international dealer.
But it’s hard to overstate how difficult the difficulty of moving off legacy systems is. As Ryan Lynch of Lafferty Group put it:
The banks would like to move to new systems but it would be like trying to change the engines on an aeroplane while it is in flight. Because of the need to back up data continuously while processing payments, calculating interest on a daily basis and so on it is almost impossible to stop everything to migrate to a new system
So the banks keep putting more and more layers of new applications and interfaces on the same creaky core, creating even more interdependencies to sort out. It’s not clear how this ends but the odds are high the end won’t be pretty.