After we’ve been writing about the problem of the ticking time bomb of bank legacy systems written in COBOL that depends on a shrinking pool of aging programmers to baby them for now nearly two years, Reuters reports on the issue. Chuck L flagged a Reuters story, Banks scramble to fix old systems as IT ‘cowboys’ ride into sunset, which made some of the points we’ve been making but frustratingly missed other key elements.
Here’s what Reuters confirmed:
Banks and the Federal government are running mission-critical core systems on COBOL, and only a small number of older software engineers have the expertise to keep the systems running. From the article:
In the United States, the financial sector, major corporations and parts of the federal government still largely rely on it because it underpins powerful systems that were built in the 70s or 80s and never fully replaced…
Experienced COBOL programmers can earn more than $100 an hour when they get called in to patch up glitches, rewrite coding manuals or make new systems work with old.
For their customers such expenses pale in comparison with what it would cost to replace the old systems altogether, not to mention the risks involved.
Here’s what Reuters missed:
Why young coders are not learning COBOL. Why, in an era when IT grads find it hard to get entry-level jobs in the US, are young programmers not learning COBOL as a guaranteed meal ticket? Basically, it’s completely uncool and extremely tedious to work with by modern standards. Given how narrowminded employers are, if you get good at COBOL, I woudl bet it’s assumed you are only capable of doing grunt coding and would never get into the circles to work on the fantasy of getting rich by developing a hip app.
I’m sure expert readers will flag other issues, but the huge shortcoming of COBOL is that there are no equivalent of editing programs. Every line of code in a routine must be inspected and changed line by line.
How banks got in this mess in the first place. The original sin of software development is failure to document the code. In fairness, the Reuters story does allude to the issue:
But COBOL veterans say it takes more than just knowing the language itself. COBOL-based systems vary widely and original programmers rarely wrote handbooks, making trouble-shooting difficult for others.
What this does not make quite clear is that given the lack of documentation, it will always be cheaper and lower risk to have someone who is familiar with the code baby it, best of all the guy who originally wrote it. And that means any time you bring someone in, they are going to have to sort out not just the code that might be causing fits and starts, but the considerable interdependencies that have developed over time. As the article notes:
“It is immensely complex,” said [former chief executive of Barclays PLC Anthony] Jenkins, who now heads startup 10x Future Technologies, which sells new IT infrastructure to banks. “Legacy systems from different generations are layered and often heavily intertwined.”
I had the derivatives trading firm O’Connor & Associates as a client in the early 1990s. It was widely recognized as being one of the two best IT shops in all of Wall Street at the time. O’Connor was running the biggest private sector Unix network in the world back then. And IT was seen as critical to the firm’s success; half of O’Connor’s expenses went to it.
Even with it being a huge expense, and the my client, the CIO, repeatedly telling his partners that documenting the code would save 20% over the life of the software, his pleas fell on deaf ears. Even with the big commitment to building software, the trading desk heads felt it was already taking too long to get their apps into production. Speed of deployment was more important to them than cost or long-term considerations.1 And if you saw this sort of behavior with a firm where software development was a huge expense for partners who were spending their own money, it’s not hard to see how managers in a firm where the developers were much less important and management was fixated on short term earnings targets to blow off tradeoff like this entirely.
Picking up sales patter from vendors, Reuters is over-stating banks’ ability to address this issue. Here is what Reuters would have you believe:
The industry appears to be reaching an inflection point, though. In the United States, banks are slowly shifting toward newer languages taking cue from overseas rivals who have already made the switch-over.
Commonwealth Bank of Australia, for instance, replaced its core banking platform in 2012 with the help of Accenture and software company SAP SE. The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).
Accenture is also working with software vendor Temenos Group AG to help Swedish bank Nordea make a similar transition by 2020. IBM is also setting itself up to profit from the changes, despite its defense of COBOL’s relevance. It recently acquired EzSource, a company that helps programmers figure out how old COBOL programs work.
The conundrum is the more new routines banks pile on top of legacy systems, the more difficult a transition becomes. So delay only makes matters worse. Yet the incentives of everyone outside the IT areas is to hope they can ride it out and make the legacy system time bomb their successor’s problem.
If you read carefully, Commonwealth is the only success story so far. And it’s vastly less complex than that of many US players. First, it has roughly A$990 billion or $740 billion in assets now. While that makes it #46 in the world (and Nordea is of similar size at #44 as of June 30, 2016), JP Morgan and Bank of America are three times larger. Second, and perhaps more important, they are the product of more bank mergers. Commonwealth has acquired only four banks since the computer era. Third, many of the larger banks are major capital markets players, meaning their transaction volume relative to their asset base and product complexit is also vastly greater than for a Commonwealth. Finally, it is not impossible that as a government owned bank prior to 1990 that not being profit driven, Commonwealth’s software jockeys might have documented some of the COBOL, making a transition less fraught.
Add to that that the Commonwealth project was clearly a “big IT project”. Anything over $500 million comfortably falls into that category. The failure rate on big IT projects is over 50%; some experts estimate it at 80% (costly failures are disguised as well as possible; some big IT projects going off the rails are terminated early).
Mind you, that is not to say that it is impossible to move off legacy platforms. The issue is the time and cost (as well as risk). One reader, I believe Brooklyn Bridge, recounted a prototypical conversation with management in which it became clear that the cost of a migration would be three times a behemoth bank’s total profit for three years. That immediately shut down the manager’s interest.
Estimates like that don’t factor in the high odds of overruns. And even if it is too high for some banks by a factor of five, that’s still too big for most to stomach until they are forced to. So the question then becomes: can they whack off enough increments of the problem to make it digestible from a cost and risk perspective? But the flip side is that the easier parts to isolate and migrate are likely not to be the most urgent to address.
1 The CIO had been the head index trader and had also help build O’Connor’s FX derivatives trading business, so he was well aware of the tradeoff between trading a new instrument sooner versus software life cycle costs. He was convinced his partners were being short-sighted even over the near term and had some analyses to bolster that view. So this was the not empire-building or special pleading. This was an effort at prudent management.