We’re turing again to TSB’s botched effort to transfer all of its users to a new computer system over the past weekend, where the bank is still bleeding from a major artery.
The TSB Train Wreck Continues
In yet another effort at porcine maquillage, the CEO today tried claiming everything was fine.
Our mobile banking app and online banking are now up and running. Thank you for your patience and for bearing with us.
— Paul Pester (@PaulPester) April 25, 2018
That didn’t last long. Again from Paul Pester:
The challenge we are facing at the moment is that while we know everything is working, one of the main ways that our customers see everything is working – through our internet banking and mobile app – isn’t functioning as well as it should be, and for this I’m truly sorry. I can appreciate how frustrating this must be for our customers.
As Financial Times reader Paper Chase remarked: “The level of contradiction in this statement is beyond my comprehension.”
The bank later stated that only about half the online customers could access their accounts, as if the only problem was now capacity. Per the Guardian:
TSB said internet banking was operating at 50% of capacity, which means that for every 10 customers only five will be able to access this service. Mobile banking was operating at around 90% of capacity, the bank said in a statement issued at about 4pm.
It appears the bank has a different idea of what “operating” means than customers do. Twitter was rife with complaints from users that while they could finally get into their accounts, they saw alarming errors and/or had difficulty performing transactions. These tweets are from the late afternoon and early evening UK time:
— Cheryl Ewing (@shallotohgravy) April 25, 2018
#tsbfail ERROR_SISTEMA showing at 18:01 so still cannot pay my Utility Bills! I hope the Utility Companies appreciate this is TSB and not me !!
— Mary George (@marygeo6) April 25, 2018
@PaulPester @TSB TSB are complete liars! I’ve been on hold trying to get though for 45-50 mins now and keep getting cut off! Don’t tell you have a problem with your phone lines now? I still can’t make a transfer on my mobile banking app! Sick to death of it now! #TSB #TSBFAIL
— Anne Morton (@amorton66) April 25, 2018
Oh @tsb, to say these problems are down to ‘bandwidth’ really sounds dubious. And the lack of incident management on this is almost unbelievable. As someone who works in IT I’m horrified at how this has been handled! #tsbfail
— julie ballantyne (@julieballantyne) April 25, 2018
It's getting worse as the day goes on. I can't login online at all now!! I'm locked out of the app as its trying to text me on a number I changed 3 years ago. To say the system is running is complete bollox @PaulPester #tsb #tsbfail pic.twitter.com/6AlLQlOR6d
— Jobseeker (@Anonymous_Nottz) April 25, 2018
Needless to say, the potential damage goes beyond late bill payments and inability to access funds, which bites for stay-at-homes who can’t go to a bank branch and people with large expenses in coming days, such as for a wedding. CEO Pester also claimed that TSB’s “engine room” was working fine and that scheduled payments and transfers were all being processed.
Given that the top management of TSB was high-fiving a successful launch over the weekend, its ability to make accurate self-assesments seems pretty impaired. As Richard Smith and some parties on Twitter noted, there are signs of data mapping and potentially even data corruption, such as some customers getting logged into other individual’s accounts, quite a few reports of mortgages disappearing or having incorrect information. These complaints in the Guardian also contradict Pester’s cheery reassurance that transactions are being processed correctly despite appearances otherwise:
Small businesses were unable to pay salaries or manage transactions, while some account holders found all their direct debits had disappeared and others reported that their cards were declined when shopping.
And Clive pointed out earlier in the week that Pester’s own remarks acknoweldged that some, Lord only knows how much, customer data has been lost:
But the bigger picture which is obscured by the “helping us to help you” advice from the CEO is — excuse me — WTF is a bank which cannot maintain an accurate ledger or record of customer product holdings? And Pester’s statement is tantamount to an admission that, for some correcting entries, they are going to have to rely on a customer telling them what they (TSB) need to go away and put right rather than the bank being able to reconstruct a valid account history, apply misposted transactions or reunite orphaned product holdings with their rightful owners.
I for one have a rough idea about what should or should not have been posted to my account — but I don’t keep a comprehensive shadow set of records. If TSB has so broken its CRM and accounting data that it needs customers to tell it what’s missing then customers will have lost money for sure — or else have credits on their accounts or be assigned ownership of products which aren’t really theirs which they might inadvertently draw on only to find out much later down the line they have to reimburse the bank because they weren’t entitled to the funds.
Richard Smith flagged this as one of several examples of a “spectacular example of a data mapping problem”:
#TSB I can't get through to customer services, been cut off after 15 mins on hold. My user ID had been swapped with my memorable info and my password isn't recognised so tried to change it and told new one can't match the old? It's been 5 days & I can't eat/travel I need my money
— sarah clark (@sarahcl42953252) April 25, 2018
I would assume that data mapping problems don’t often occur in isolation.
So TSB has data integrity problems. On top of that, you have the potential for hackers to make off with funds or do damage to records while trying. From the Financial Times:
In its Sunderland call centre, stressed staff were offered free fruit to keep their spirits and vitamin C levels up as they dealt with thousands of angry customers. Two people with knowledge of the situation said workers lacked confidence in the security checks intended to weed out fraudulent attempts to take advantage of the confusion. Several staff walked out.
Let’s step back a bit and see what this says about the state of banking.
How Did This Screw-Up Happen?
On the one hand, as the sorry story of TSB’s cack-handed effort to move to a new system is likely to show, that Banco Sabadell, which bought TSB in 2015, is particularly incompetent in managing bank systems. On the other hand, as we’ve discussed before, big IT projects regularly fail, with the admitted abort rate at about 50% and the actual probably closer to 80%. Yet the way banks have been running their systems for decades, by not making adequate documentation and investments, seems virtually guaranteed to set them up for huge train wrecks.
Signs that Banco Sabadell is particularly clueless even by the low standards of bank IT managers. Since regulatory and Parliamentary proctology is virtually guaranteed, we’ll know more about TSB/Banco Sabadell’s pathologies in due course. Nevertheless, some of the statements of bank executives, combined with examples of error messages, demonstrate that the Banco Sabadell top brass is particularly clueless. Aside from the repeated premature declarations that everything was fine or nearly fine when it wasn’t, consider:
This acquisition should never have been made. In the US, we’ve had way more bank mergers than anywhere in the world by going from a highly fragmented banking system (over 16,000 banks in the late 1980s, and even that number reflected some mergers in during that decade, to about 6800 in early 2014). That much trial and error means financial services industry executives have learned what not to do. One is to pass on a bank acquisition when the systems integration looks hairy.
TSB was a mid-sized bank deal, with £42 billion in assets at year end 2017, or about $60 billion, and roughly 550 branches. By way of comparison, Canadian bank TD ate my somewhat larger-than-TSB-bank, Commerce Bank, in 2008. The only IT hiccup I’ve seen is that they too often make a mess of wire transfers.
There are some red flags that Sabadell and TSB systems had major compatibility issues. The first was that TSB’s former parent Lloyds had entered into the unusual arrangement of running the TSB systems for Sabadell for years after the 2015 sale. Second was Sabadell’s bold pronouncement that it was going to move TSB customers to a “brand new core banking system“.
The timetable as insane.The Financial Times published this astonishing (or maybe not in light of what happened) tidbit:
A person briefed on the TSB board’s plans insisted the platform had not been rushed out, and said questions over who was responsible would have to wait until the system was running smoothly for all its customers: “Given the testing they didn’t think this was where they were going to end up — they never would have pushed the button otherwise,” the person said.
Huh? It’s not exactly clear when Banco Sabadell took operational control of TSB. Even though Sabadell acquired all TSB shares on August 21, 2015, an August 29, 2015 Guardian story describes Lloyds having to divest TSB branches on behalf of Sabadell, with the headline TSB to shut 17 branches before takeover by Spanish banking group Sabadell and says in the body that the deal was yet to be completed. Sabadell would not be asking Lloyds to handle divestitures if closing were in days or even weeks. So our assumption that Sabadell took operational control at the end of September 2015 would be generous.
That would mean two years, seven months for this “brand new core banking system” to be built. Contrast this with vlade’s comment on project timelines:
Just a version migration of the same system was easily a year/two programme. A new system introduction to replace a legacy system was 3-5 years easily..
I was also involved in post-sale migration at the time, and it was also a 18months + programme, even if it was rather simple (as in maybe a few thousands clients, and few tens of thousands transactions were being moved).
I’ve also seen a replacement of the retail system done (but wasn’t directly involved), and again, it was a pretty few years worth. Customer base would have been about the same size as TSB.
It looks at if Sabadell was sold more than a bit of consultant hopium. From Financial Times reader Stephen:
Apparently Banco Sabadell’s Proteo is based on Accenture’s alnova banking package, which is an old COBOL system modified to run on Unix under .NET. Sabadell’s new Proteo4UK version of it runs in Amazon cloud.
Amusingly, Sabadell released a self-congratulatory press release on Sunday afternoon, now preserved for posterity in Google’s cache:
Banco Sabadell successfully completes TSB technology migration…
For TSB, the Proteo4UK migration project is the best strategic decision it could have made when compared to the technological alternatives considered at the time, as the platform fulfils its business requirements and supports its strategy. Total synergies arising from the migration are estimated to amount to £160M on an annual basis.
The new Proteo4UK platform offers increased operational simplicity, which, for example, enables significant time savings in the implementation of the principal business and operational processes. The new platform also generates new opportunities, such as the launch of the SME business and enhanced user experience thanks to the digitalisation and homogenous omni-channel deployment of products and services.
Testing was grossly inadequate to non-existent. Richard Smith, without looking hard, found ready evidence, such as BeanCreationException. More generally, to IT pros, the fact that Twitter is full of screenshots of customers actually seeing gnarly error message is a gasp-out-loud programming failure. As our Otis B Driftwood said yesterday:
Those TSB messages violate a cardinal rule of software development to NOT expose unexpected internal error information to an end user. They should never have passed design and code review and QA testing.
The errors revealed through these messages should never have passed unit testing in the first place. Accessing an array out of its bounds? Dereferencing a null pointer? Attempting to use a connection that is not set up?
All those circumstances indicate that the programmers do not initialize their variables properly, or are using them in ways that get them overwritten / deallocated between procedure calls — and in addition do not even attempt to check for abnormal situations.
Or perhaps they are relying upon a third-party application framework that is riddled with those problems.
What Does This Ciusterfuck Say About Bank IT?
Given the discussion above, we are likely to learn officially that Sabadell did every bit as lousy a job of managing this migration process as we can infer from our remove. But our hypothesis that this would have been beyond the ability of even a much more technologically competent bank to execute.
Several readers pointed to the fact that RBS allegedly spent over five years and £1.5 billion in a failed effort to clone its systems so it could divest roughly 300 Williams & Glyn branches. That should have been easier than what Sabadell was trying to execute with TSB. As Financial Times reader SpeedFerret noted in early 2017:
Who in their right mind would buy a stand up bank of just 300 branches with a sprawling technology stack of over 600 (yes 600) ancient applications to support it. A very poor decision to come up with this strategy in the first place. £1.5bn thrown to Indian IT offshore consultancies to untangle the spagetti of systems and overnight batch jobs came to nothing. A further £750m to be handed out as dowries. All we need now is the ATM network to be out of action for a week over the Easter holidays.
That gives a tiny vignette into the real problem, which we hope to chip away at in later posts, which is that bank systems are a greater mess than pretty much anyone on the outside imagines and are kept running with liberal applications of duct tape and baling wire.
The very short version of “How could banks have possibly gotten themselves in this mess?” is that IT has never been treated as mission critical when it is. Banks have underspent in the systems area for decades. The biggest symptom is lack of documentation. It costs money and adds to project completion time (although it reduces life cycle costs, but who cares about the long term?). The second is rushed projects with bad code in them that only kinda-sorta works. With systems in production, it’s almost always perceived to be too costly to identify, rip out, and redo the botched parts. Instead, the IT pros resort to patches and work-arounds. That adds to complexity and fragility, particularly when the system gets stressed or large changes are made.
The problem of systems management might be a smidge worse in the UK than the US if the erosion in the managerial capabilities of what passes for the elite that we see in its political classes has occurred in paler form in the private sector. Clive has spoken of a style of managerialism that holds specialist knowledge in contempt and views people that try to point out why certain goals are unrealistic as not clever enough. Mind you, we have that in the US thanks to management by MBA, but the rot may be less advanced, since most MBAS recognize that they are exploiting underlinings with expertise and posture that their skills really don’t matter much, while their British counterparts may believe that on a much more visceral level.