Twenty banks (some suffering repeated outages), six countries (one in lockdown), five continents, tens of millions of unhappy customers.
There’s never a good time for your bank’s IT system to go down. But few can be worse than in the middle of a lockdown. It’s difficult to leave home, your local branch may not be open, and as a result you are more reliant than ever on digital banking services. In New Zealand, now in its seventh week of nationwide lockdown, one of the country’s largest lenders, Kiwibank, went down on Tuesday, leaving many of its customers in the lurch. It is one of a string of IT outages the bank has suffered over the past three weeks, after a DDoS attack on New Zealand’s third largest Internet provider caused IT crashes at a number of lenders, including Commonwealth Bank and Anz Bank.
In a DDoS attack hackers overwhelm a site by getting huge numbers of bots to connect to it all at once, rendering it inaccessible. Servers are not breached, data is not stolen but it can still cause plenty of disruption.
24 Million Unhappy Customers
New Zealand is not the only country to have suffered major outages within its banking system in recent weeks. Other countries include the UK, Japan, South Africa, Venezuela and Mexico, though there are no doubt more (if you know of any, It would be great if you could provide details in the comments section).
On September 12, operating failures at Mexico’s largest bank, BBVA Mexico, left 24 million account holders unable to use the bank’s 13,000 ATMs, its mobile app or in-store payments for almost 20 hours. It being a Sunday, customers could not even avail of the lender’s in-branch cash services. The bank blamed the outage on a system update failure and has offered to compensate customers with cash bonuses on purchases when using their credit or debit cards. The bank was also at pains to assure them that their financial data was not compromised.
“It had nothing to do with the outside world,” said Jorge Terrazas, the bank’s director of communicate and corporate identity. “The bank and its customers’ information is secure. What we did was undo the changes to the system and return everything to as it was.”
Less than a week after BBVA’s outage, Santander Mexico, another Spanish-owned Mexican bank, suffered an outage that left customers across the country unable to use their debit cards at the ATM or in stores. Again, it was blamed on internal problems.
In recent years, Mexico has become an important market for stolen data — enough to earn it eighth place in the world in terms of identity theft, according to the country’s central bank, Banco de Mexico (Banxico for short). This is partly a result of the widespread impunity cyber criminals enjoy in the country, due to the lack of enforcement of existing laws and the absence of adequate legal tools. Cyber theft in Mexico is not just the preserve of isolated basement-dwelling hackers but also highly professional criminal organizations.
Even Banxico’s SPEI interbank transfer system, an iteration of the SWIFT global payment system, has been the target of digital heists, as WIRED reports:
In January 2018 a group of hackers, now thought to be working for the North Korean state-sponsored group Lazarus, tried to steal $110 million from the Mexican commercial bank Bancomext. That effort failed. But just a few months later, a smaller yet still elaborate series of attacks allowed hackers to siphon off 300 to 400 million pesos, or roughly $15 to $20 million from Mexican banks. [Click here to read how they did it].
Since then Mexican banks have suffered repeated outages, one of the biggest of which took place during last year’s “Buen Fin”, an annual nationwide shopping event inspired by Black Friday. The online banking websites and mobile apps of many of the country’s major banks, including BBVA and Citibanamex collapsed on the same day, leaving many customers unable to complete their purchases.
“A Growing Trend”
In the UK the Financial Conduct Authority has been “deeply concerned” about the increasing number of technology outages for a number of years. At the FCA’s annual public meeting in 2019, the regulator’s executive director of supervision, Megan Butler, said the number of incidents of “operation resilience breaks” reported in terms of IT failings had increased 300% year-on-year. And this, she said, would probably be “a growing trend,” though it is partly due to the rise in reporting of events.
On July 22 this year, the websites of six large banks and building societies — Lloyds, HSBC, TESCO Bank, Bank of Scotland, Halifax and Barclays — were brought down by a global Internet outage allegedly caused by a botched software update at hosting service Akamai. Less than a month later, the apps of five lenders and building societies — Natwest, TESCO Bank, TSB, Santander UK and Halifax — all went down over a period of just a few days. The outage, apparently triggered by a problem with US payments company TSYS, left consumers unable to access their credit card services and account information. Since then, HSBC, Barclays Bank and the Cooperative Bank have all suffered brief outages.
Some outages can last much longer and wreak far more disruption on people’s lives. In 2018 Banco Sabadell’s botched IT migration of its UK subsidiary TSB — branded the “biggest IT disaster in British banking history — left hundreds of thousands of customers unable to access their online accounts for weeks on end. Some customers lost out financially. Many saw their credit ratings deteriorate as a direct result. Business customers were unable to pay bills or make payroll and mortgage payments were missed. Over 1,300 customers became victims of fraud attacks. The crisis cost Sabadell hundreds of millions of pounds, 80,000 customers and one CEO. It was probably a key factor in scuppering BBVA’s takeover of Sabadell late last year.
“An Intense and Aggressive Cyber Attack”
Almost 5,000 miles away from the UK, on the other side of the Atlantic, 16 million customers of Venezuela’s biggest bank, Banco de Venezuela, had to recently endure five days without the bank’s online platform. As tends to happen in these cases, the outage became apparent when bank customers began venting their anger on social media. When the platform was finally restored, on September 20, Venezuela’s vice president Delcy Rodríguez laid the blame on the US government, which she accused of launching an “intense and aggressive” cyber attack against the bank’s IT system.
The attack was apparently an attempt to derail Caracas’ plans to launch a new currency, which went live today (Oct 1) with six fewer zeros. Whether Rodríguez’ allegations are true or not it’s impossible to tell, but Washington certainly has the capability and form. Plus, it is engaged in a no holds-barred economic war against Venezuela.
Sometimes it’s the frequency rather than the duration of the outages that is the biggest problem for bank customers. Yesterday (September 30) Mizuho Bank, one of Japan’s three mega banks, experienced its eighth IT system failure so far this year — almost one every month. In the latest episode a system glitch caused a delay to some foreign exchange transactions. The system outages at Mizuho date back almost two decades and have been broadly blamed on its failure to integrate cultures and systems from the three-way merger of Dai-ichi Kangyo Bank, Fuji Bank, and IBJ that brought the bank into existence, all of 21 years ago. The bank has already spent $3.6 billion trying to fix the problems, but to little apparent avail.
Mergers of large banks have a tendency of leaving behind serious IT system issues, as Clive and I pointed out in an NC article published in December last year. This is particularly true in the case of cross-border mergers. One of the main reasons for this is that many banks are still largely run on creaky legacy systems built in the 1970s that make it all but impossible to merge IT systems without storing up big problems further down the line. In a 2019 Treasury Select Committee inquiry into what went wrong at Banco Sabadell, Alison Barker, director of specialist supervision at the Financial Conduct Authority, was asked to what extent legacy systems are still being used across the UK’s retail banking sector. Here’s what she said:
“It is still pretty extensively, I’m afraid… some pretty core systems are still run on legacy. They still use code back from the 1970s on some of these systems, and they’ve just built on top of them.”
Yet many of these same banks are still trying to compete with younger, smaller, fleet of foot challengers whose IT systems are much more modern and flexible. And that is causing serious problems.
Inherent Fragility of Legacy Systems
“If you are a large retail bank in the UK, you are probably dealing with legacy systems”, the deputy chief executive of the Prudential Regulation Authority, Lyndon Nelson, told the inquiry. But as fintech companies add new features to their apps, they are keen to do the same “for competitive reasons.”
Nelson added that although some banks do plan to eventually phase out their legacy systems, it takes a brave chief technology officer to envisage that, due to the inherent risk in changing systems. Sabadell’s disastrous attempt to upgrade TSB’s system will hardly have encouraged others to do the same. As S&P Global recently noted, bungled IT change is a leading culprit for outages and disruptions at U.K. financial institutions. An overreliance on outsourcing could make the problems worse.
Another problem highlighted by Nelson (and NC way back in 2016) is that few programmers are left who can actually use COBOL, the primary programming language used in banks’ legacy systems. This, says Nelson, has left many banks’ IT officers asking the question: “how many times in a week can we change an app without it falling over?”
When a banking app “falls over” or an IT system goes down, it can leave chaos in its wake. Ten years ago, Mizuho Bank suffered an outage that delayed money transfers in the aftermath of the Great East Japan Earthquake and tsunami. Its seventh outage this year, in early September, was apparently the final straw for Japan’s financial regulators, which requested that Mizuho submit a work plan for system maintenance and updating, “in a rare move to effectively oversee the system of a megabank”, reported Kyodo News.
Another bank that has been plagued by repeated IT system problems is South Africa’s largest lender, Standard Bank. In late April, the bank suffered “hardware issues” that downed its internet, mobile and ATM channels for over a week, leaving customers unable to pay their bills or access cash. By early September the bank’s mobile app was down once again, causing customers no end of hassle. On Tuesday this week the mobile app of another South African bank, Capitec, also went down.
All of these bank outages are happening for a variety of reasons, from internal problems within a bank’s IT system (Mizuho, Sabadell) to a botched update (BBVA), to a cyber attack (Kiwibank), to the downing of a hosting service (the collapse of bank websites around the world on July 22). But one thing they all highlight is the inherent fragility of banks’ IT systems, at a time when many people are using less and less cash and are becoming more and more dependent on digital banking services.
30 comments
Comments are closed.
This post just confirms my belief that it never hurts to keep sone good old fashioned cash around ( until they try to outlaw that). E-banking goes dark at least you can get some food to survive for a little while. Gas up the truck.
Except that when e-money goes poof, the price of everything goes up a jazillion per cent. So unless you’ve managed to squirrel away several million in cash – small bills only, please – instead of a few thousand or whatever, you’re as screwed as everyone else. Maybe the trick is buying gold?
Gold would only be useful for large transactions. Silver is more feasible. However, look at the state of the silver market. Try to get quick delivery of physical silver. There is a “shortage” of physical silver available.
Of course, as any “hard currency” aficionado will tell you; the silver market has been manipulated by large financial organizations for years.
Another fun fact is that, if the “average” American worker cannot come up with $400 USD in an emergency, how are they going to aford the roughly $200 USD that Ten dollars face value of “junk silver coins” will set them back right now? The actual price of the silver metal in the coins in that Ten dollars face value is presently about $150 USD. Add to that the 30% to 40% “handling charges” the commercial silver dealers pile on top of every transaction, buying and selling, and you end up with a system designed to keep out the “average jane or joe.”
As currently constituted, the “hard currency” regime is a very regressive tax on the working class.
Why would prices go up when the e-money disappears? And how long would they stay up for?
During the Great Depression, didn’t a lot of that time’s equivalent of e-money disappear? And what did prices do then?
And ask merchants for a cash discount. They often pay at least 4% to credit card companies, so they should be willing to at bare minimum, split that and when the paperwork complications are added, maybe more than that as a discount.
Ah yes, a reference to lack of COBOL programmers. Seen that before, I think for a different sort of legacy systems, state unemployment websites. I can’t imagine modern CS instructors are equipped to instruct classes on that language.
But then again, its not Fortran! I took courses in both program languages long ago. I passed the courses, about the only success I can point to.
I can tell you the NYS unemployment system website is archaic. It literally looks like something a high school computer club in the 90’s would have dreamed up. It doesn’t even keep a record of your messages that you send in seeking the money you are owed. That’s probably a feature.
Incidentally, all the other state sites I used seemed much more up to date. Funny…
A similarity between this article and the Atlantic piece about the next pandemic: legacy systems that are not up to the task. Banks running on 1970’s software; public health infrastructure from the 1940’s and 50’s. A lack of investment and no vision of the future.
But don’t worry about the bankers, they are safe, quite safe. A family member’s close friend just joined the security team at a big financial institution. Bomb/bullet proof buildings, on-site medical staff, and a small army of highly trained goons. This is just in one building in NYC. I’m sure they won’t ever lose access to their funds either and they sure as hell won’t be riding the train when the next pandemic hits.
The problem with that strategy is that Timmy McVeigh and his co-conspirators showed how to defeat it.
Indeed he did. Come to think of it, I also wonder how their tech-heavy defenses will fare when a winter storm blows out the power and some hacker keeps it down. Generators for a while but when the fuel sours or runs out. I’d bet those armories are electronically keyed.
Greer wrote a piece a while back about how the wealthy families in the Hamptons hired private security forces to keep the proles away when trouble threatened. When the SHTF full bore, scavengers found McMansion after McMansion filled with corpses, a hole in each of their heads.
I wonder if healthcare systems from the 40s and 50s might be better than what we have now. Since the 1970s the USA has lost a large amount of hospital beds thanks to the consolidation of health care, and many small hospitals in rural areas have closed.
I bet that Covid was the straw that nudged the backs of the old-school programmers to retire, regardless of telework or bonuses.
just as all the institutional memory from building the Saturn V evaporated, so will COBOL, etc
Based on a previous article, that memory will especially vanish if it’s stored in files that are in folders.
I’ve thought about that story where they say younger people have problems with the concept of a folder for files.
Step outside of the computer world and think of how a folder is used. It’s used to put personal or PRIVATE things in. That’s the concept that has to exist to understand “why a folder?”
yes, COBOL data are files in folders.
Funny thing – the “newest” technology for big data does guess what? Store files in folders.. (admitedly, it’s a way that COBOL could not even dream of, but the fundamental principle of folder/file is more or less the same)
Nope.
My brother is a Cobol prorgrammer who just turned 65, overseeing a mission critical legacy system. He loves his job. He loves his immediate manager. But the department head is a sterling example of the Peter Principal at work. A brilliant programmer with zero social skills, the youngest department head ever hired by the organization. Ageist to her Python-centric core. If her older employees were minorities and she treated them with the same attitude she treated her older employees, she would be out on her ear in a week.
My brother himself is fleeing in January.
As the other legacy programmers flee in droves, higher management blames WFH, when it fact, that was all that was keeping them on the job in the first place.
Are they just following the science?
Here in the US, I am in my 80th week of not going to the gym, night clubs, political meetings, theater or indoor parties because I like my lungs, heart and brain too much to pretend that this isn’t real.
I envy the New Zealanders.
Except it looks like Delta was too much for them too, as the lockdown from August is still (in Auckland) in force and the numbers aren’t coming down.
If NZ with their advantages can’t eliminate it, I’m not sure any non-police state can.
What would a virtual bank run look like?
Only the banking institution would know depositors were taking out their money en masse, so there wouldn’t be the usual panic that ensues when the line goes round the block, in old school runs.
When you can’t access your bank from the app on your phone, or from an ATM, it does not mean that the ‘legacy’ systems at the bank have failed you.
It’s probable that those legacy systems are humming along just fine, but the conglomeration of systems that allow internet-based services that must interface with the legacy system at the heart of your bank have a failure at some point.
The internet evolved so fast that there was no refusing to “go along” by anyone, including your bank.
When it became possible to replace dedicated terminals with PCs, it happened overnight and those PCs were in most cases were soon attached to the internet.
I was working in IT at the time, and the speed with which all this happened is hard to describe.
Then came the fast-talking marketers of the Dot-Com period, who promised a whole lot of stuff that was entirely possible, but that they had no way of delivering because they hired salespeople, but not engineers.
Then came the phones and social media, and apps and the fast-talking marketers selling seamless integration of systems that by their nature resist seamless integration.
It’s not the fault of ‘legacy ‘ systems and COBOL that the sh*t doesn’t work, it’s the Rube Goldberg pyramid that’s been grafted on to them, and the MBA culture that never asks enough questions before signing the contract for the Next Big Thing.
And now, when the Next Big Thing has become both essential and unreliable, we’ve been invited to believe that it’s due to the fragility of legacy systems.
That excuse, and it is an excuse, is fundamentally dishonest, and while it may be true that my rant isn’t helping the situation, neither is making misleading excuses.
The situation with TSB in the UK being aquirred by Sabadell is a perfect example, from Wolf Street;
And there’s the assessment by IBM who was brought in to fix the mess;
None of the above constitutes proof that legacy systems are fragile, on the contrary, it places the blame on bad decisions by human beings.
Extremely interesting, thank you
I work in IT for a largish dot com and everyone is pretty competent, but the amount of stuff that is held together with duct tape and glue would surprise a lot of people.
And if you see what I see, that duct tape and glue is applied at the behest of management because they want to avoid the cost of the solution proposed by their IT department.
In some cases, the IT department doesn’t know what the proper fix is, and would like some outside help in assessing the proper fix, this sort of help comes at additional cost of course, so management asks if some duct tape and glue would work, and somebody in IT says “maybe”. (If the duct tape holds for the rest of the quarter, that guy gets promoted.)
From the managements perspective, the duct tape and glue Rube Goldberg result is superior due to its lower cost, and coupled with the IBGYBG culture so prevalent in management, the situation we’re discussing is present everywhere.
Thank you that really explains everything.
As a KiwiBank customer I was mildly inconvenienced by the outage but I can no more blame the Bank for being on the receiving end of a DDOS attack than I can someone who was mugged for walking down the street. Discussing it with others I received the impression that most people were like me in believing it better that Banks hold the line against these blackmailing scumbags, as paying up to make them go away just exacerbates the problem. I know my and I suspect the opinions of others would be different if the problem had been a result of internal incompetence and/or trying to run a high-tech system on the cheap.
That said, KiwiBank ‘phased out’ the use of cheques earlier this year, closing the door on the fall-back option of popping round to the bricks-and-mortar bank and withdrawing a wad of cash for use when EFTPOS isn’t working.
The desire to replace the local bricks-and-mortar banks with far cheaper electronic banking is driving its promotion, but the fragility of the latter illustrated by these events should – but probably won’t – suggest to Governments the need to bang a few Banker’s heads together to co-operate to retain some bricks-and-mortar cash outlets in the High Street, worked by people with pens and paper.
Earlier this week BBT, now Truist was down. I couldn’t log into the online banking portal. I checked a couple of days later and it was still down. I haven’t checked today. Hopefully it is working.
GASP, the ayn rand bank! oh the scandal:)
Yes, this is like someone being mugged walking down the street… through the bad part of town, at night, while waving handfuls of cash and yelling ‘look at all this money!’
This is a natural consequence of putting critical infrastructure on a public playtoy like the internet.
Just wait until the all mega’s consolidate in AWS and regionals in FIS core banking platforms. No concentration risk there. /s