Yves here. Based simply on Clive carefully drawing out the implications of public information and a wee bit of testing on his own TSB account, it is becoming more and more apparent that the IT disaster at TSB is not only vastly worse than the bank maintains, but was completely self-inflicted. It’s as if they decided to play Russian roulette with a gun with a bullet in every chamber.
Clive’s forensics demonstrate:
Basic customer information is not mapping to his account. TSB does recognize that he had a credit card and apparently has its credit line correct as well. But it has no transaction history, sees it as a new card when it isn’t.
New transactions are not being associated with his account either.
The systems problems aren’t simply an “online banking problem” and go well beyond that. TSB and its parent Sabadell have been telling customers that the “engine” is fine, they just can’t get those pesky, flaky apps to connect to that humming engine properly. Clive went to the branch and they couldn’t find his new transaction either.
Twitter complaints indicate that some (many?) customers who have mortgages similarly have the TSB system, when they finally get in, showing a zero balance. That says the problems with finding historical data are affecting a yet undetermined number of customers for at least two major products, and the ones that almost certainly account for most of TSBs credit risk and profits.
In a best case scenario, TSB has a data mapping problem that it presumably can eventually sort out, but the cost and time involved could be very high. In a worst case, the data is corrupted or lost. And if that has happened on anything more than a trivial scale, I’m not sure how the bank recovers from that. Similarly, if the bank is making payments but they are floating in some sort of data purgatory not posted to account, what does it take to clean that up?
Contrast that with the continued delusion back at Sabadell, courtesy the Guardian:
The bank’s boss, Paul Pester, said TSB will waive £10m in overdraft fees and pay extra interest on current accounts. He has hired a new team of IT experts from IBM who have been told the problems must be fixed by Saturday.
Does Pester also tell pigs to fly?
Update 6:00 AM, courtesy Richard Smith. Well, at least TSB does appear not to have lost track of how much cash is in checking and savings accounts. But otherwise, as he notes: “Going great in the branches, I see.”
#TSB just went into branch and have withdrawn all my money. Have no confidence @tsb to sort this! Branch had put up barrage of desks and chairs and security guy to block access to cashier and letting people in one at a time.
— Victoria Mason (@VicMas4) April 27, 2018
So much for the cheery claims in the press that most customers won’t leave. The body language of the barricade says TSB is not so optimistic. Shorter, per Lambert; “Ooh, a bank run. What fun.”
By Clive, a UK-based IT professional
Watching the unfolding TSB IT debacle (see here and here for a catch-up if you’re late to this party) it has been depressingly predictable but no less disheartening to see the obfuscation, sophistry and media messaging war which the bank’s management has been waging. All evidence — such as the Tylenol contamination — on how to best handle such crises is that honesty and action is a far better approach than trying to spin your way out of trouble.
I happen to have a TSB product — a credit card which I use as a back-of-the wallet emergency card, just in case I need an alternative means of payment and there’s a problem with my regular cards. So I couldn’t resist using it as a test product for how well, or rather badly, TSB’s IT system recovery is going.
By way of background, there’s three cornerstones to the problems which TSB created for itself as part of its migration to a new banking platform system (and its customers). One was the migration event which needed two days to get all the legacy data over to TSB and run the import jobs — this was obvious from messages which TSB sent to its customers, such as myself, in the run-up to the switch over to the new system.
When I read the communications from TSB, there were a couple of stand-out facts which had me shaking my head. The first was that TSB were moving their entire customer base over to the new system in one go. This required a two day (the weekend of 21-22 April) complete system outage which implied it was only just about feasible to do the transfer and the data load in that timescale. This meant that there was no possibility of pre-live testing. And that there would be a single Critical Success Factor to the migration which was simply measured as getting all the data over to the new TSB system and the imports completing. Whether that data was the right data for the fields being populated in the TSB system and whether fields were mandatory or optional in terms of getting data inputted into them before the start of the on-line day on Monday 23rd was never apparently considered. There are now big gaps in the historic data on the TSB platform – I’ll cover this more below.
Once reports of the botched migration got widespread press attention, I decided to run a test of my own, to see exactly what was the truth of the situation and how that compared to what TSB was trying to convince its customers, the regulators and politicians was the current position. Having given TSB a day’s grace period to resolve what could have been fairly-easily rectified teething troubles, plus another day to live up to its promise of then fixing the issues it had admitted to I picked day 3 (Wednesday April 25th) to run a transaction through TSB’s new IT system. I used a large UK retailor with a fairly stable and mature EPoS infrastructure in order to rule out any possibility of acquirer-specific issues. I also, wisely as it turned out, generated a small-ticket value transaction (less than £20).
At the merchant, I presented my TSB credit card, which had a nil balance on it as the account is always settled in full at the end of the billing month. When I presented the card, the cashier asked if I could use “contactless” (NFC chip enabled) as there was a queue and it would be a quicker verification method than the traditional Chip and PIN, so I obliged and tried to wave my card at the card reader. After a few seconds delay, the EPoS terminal advised “Contactless Not Possible Use PIN” which was very odd. Only very rarely should contactless transactions be rejected — sometimes the card issuer or card scheme will make a random request for a PIN as a fraud prevention measure, or there’s a consecutive contactless usage limit (such as if you try to do more than 5 or 10 contactless transactions in succession). The only time apart from those narrow situations is when it is a newly issued card and it is the first occasion that card has been used, where you must initiate a PIN-verified transaction first before contactless becomes enabled. But that wasn’t the situation with this card — I’d done several contactless transactions in the past few months. So that told me something was amiss. However, after entering my PIN, the transaction seemed to go through – the receipt from the merchant and the EPoS terminal showed the amount I spent had been authorised.
The next day (Thursday April 26th) I used TSB’s online servicing channel to look at the account and see if my transaction had been posted.
I was here that I discovered the second fundamental problem which you have — the new TSB system itself. It is simply not production ready. There’s loads of cosmetic issues such as ridiculously old date ranges:
Then there’s more serious things like navigation journeys which do incorrect actions (e.g. “Ways to Pay” which should allow you to settle your credit card balance but instead is where users end up being shunted into prompts for credit limit increases):
And that then causes knock-on problems because it triggers functionality which isn’t valid but the system is to dumb to check first – in this case, the TSB system thinks the account is too new to be eligible for a limit review but it nevertheless triggered it as an option anyway.
So, returning to the point about missing data, that shows the third major defect and even deeper issues – the credit card I looked at is a minimum of two years old from the date of account opening, however the way the data was imported into the TSB system, it is being treated as a new-to-brand card dating, presumably, from less than a week ago. Once I saw what data was held about my credit card account, the reason for the contactless transaction failing was obvious — TSB’s new system is treating my two-year old (minimum) credit card account as newly-opened. My card has at least a year or more of archived statements, but when I tried to view these, the TSB system just hung – so there’s enough data been imported into the new TSB platform to tell it there’s multi-year statement history available, but not the statement history itself. This is a UK Financial Conduct Authority (FCA) regulatory breach – you can’t cut customers off from accessing their account records.
Interestingly, while the transaction amount has been earmarked from the account (it shows an available balance which is less than the credit limit on the card by the amount of the transaction) the transaction details are not yet posted to the account when I attempt to view an account statement – this should have happened overnight and the details of the transaction showing today. So while payments are “working” on the TSB system, as in, not being declined, they aren’t working end-to-end in terms of posting to the customers’ accounts from the back-end through to the front end account servicing layer.
As a follow-up, I went into my local TSB bank branch, advised the teller that I’d made a payment but it wasn’t showing on my online statement of account. I said that I had a retailer receipt, but wanted to know when or if this transaction would be posted to my account. And here I got to the most worryingly of all part of this story — the teller checked on the branch system (I specifically asked them to make a thorough check, including any pending transactions) and pronounced they could see no record of my credit card payment at all. I did ask if I should contact the retailor to make another payment if this one wouldn’t be going through, the teller advised me that customers should contact the payee and make a payment if needed. TSB simply could not confirm if the payment would be valid or not. Only time will tell if this transaction does hit my account, in the right amount with the right details.
As it was only my testing of TSB’s systems, I didn’t have an urgent need to confirm this. Imagine if this transaction had been genuinely important to either the person making, or the person receiving, the payment. Or both.
Finally, going back to the new IT system, there’s unforgivable application logic and design bugs. Just for fun, I tried to increase the limit on my credit card account. Even though I don’t have a mortgage or pay rent, the application insisted I populate this field with a non-zero value:
I tried entering nominal sums such as £10 into the field (which would be a misstatement on an application for credit, which is technically fraud) and even larger more typical sums such as £500 or £1000 a month (this would have been fraudulent too). But these were rejected with a message “there is a problem with the information given” – without telling me what the problem was. So TSB is going to get hit on the revenue side, because the sales process is trashed as a result of system issues.
During the course of my testing the TSB online system, I got thrown out several times to the “Logon” screen. This suggests that there is very short session timeouts being set to allow people to have a few minutes access on their accounts then after very small periods of inactivity, they got bumped off to make capacity available. Either that, or you kept landing on the “dead” side of the load balancer and getting thrown off the system as a result – TSB has admitted they are only up on one leg of their 50/50 load balanced system. Response times were also poor – a minute or so to navigate from page to page was not uncommon.
For TSB to say or allude to these IT system problems as being “fixed” is, as my experience and evidence above shows, at the most very, very generous reading of the facts, is a partial view of the truth. At worst, it is downright lying.
Overarching all of this debacle is the complete unsuitability of CEO Paul Pester to manage situations such as this. Infamous for a hard driving heavy on the machismo management style (one senior trade union official I spoke to who didn’t want to be identified said that Pester frequently bragged about competing in endurance sports such as triathlons to demean less physically fit colleagues), Pester is suggestive of a hypermasculinity personality disorder. The corporate culture is aggressive and reported as being frequently intimidating. TSB – and Pester in particular – is zealously anti-union with well-documented attempts to impinge on union-member confidentiality privileges. Blaming, unrealistically demanding and overly-critical management styles are guaranteed to make this sort of bad situation even worse.
No wonder no-one wanted to tell Pester the truth about how unprepared the new TSB platform was to host live customer accounts. We’re all — TSB’s customers, retail stores or similar payees and bullied, harassed and browbeaten TSB branch or call centre staff — acting as (sometimes unpaid) TSB IT testers and defect fixers as a result.
Postscript: After I had checked this post in for Yves to edit and schedule, I made a final attempt at logging in to TSB’s Internet Banking Service before turning in for the evening at around 9:15 PM UK time just on the chance my mysterious test transaction had appeared. The site was even flakier than it had been earlier in the day. When I did finally manage to log in after numerous attempts, all my account statement would show was –
An error has occurred
We’re sorry, but we’re experiencing some technical issues with the service. Please call us on 0345 835 3846. If you’re calling from abroad you can call us on +44 203 284 1581. Lines are open 24 hours a day, 7 days a week
010
This is now day 6. With no end in sight.
71 comments
Comments are closed.
Does anyone know where the actual coding was done?
I have my suspicions on the usual cost-cutting suspects, but wish to cast no aspersions.
Also, why do I suspect this is a harbinger, not a one-off?
IT very much looks like clue-less outsources. TBH, a relatively small bank like TBS will try to keep its IT dept small, so usually would not have resources to do a full migration. You know, IT folk are cost, and costs have to be managed (despite the fact that sucesfull businesses tend to focus on generating revenue first, as w/o revenu no cost management will make you profitable)
The thing is, even if your outsourcing works perfectly and your new system is delivered on time with no bugs, then after that, the team that produced it will quickly dissipate. So when you want to make any changes a couple of years down the line, the guys doing it will be completely different people from the originals.
At best that means that changes will be very expensive. At worst it means that changes will cause fundamental breaks in functionality. For companies like banks where IT is so utterly critical, outsourcing seems an awful gamble.
TSB didn’t even have an IT department, remember? Lloyds was running its systems until this migration. Sabadell is a €220 billion bank, in assets at year end, which means TSB’s £42 billion in assets is just a little less than 25% of total assets.
But the point generally is well taken. Hard to imagine that Sabadell had remotely the team to handle this scale of migration, even before you consider the evidence of the nature and magnitude of the mess, which says they had no idea what they were doing.
https://www.computerweekly.com/news/450431320/CIO-interview-Carlos-Abarca-TSB
100 people.
The use of tenses is dodgy. This is what the CIO says:
This was not the “previous situation” as of the date of the article, which was December. Lloyds was still running TSB’s systems. So it’s not clear what his bit about building the new team means, as in how much of this was in progress. And regardless, they’d need some staff to babysit the consultants building the new system (per our post yesterday).
Agree the context is not very clear, but the scale is ridiculously small. Barclays has 3500 in Radbroke that support the retail IT. This project had to be completely outsourced.
Is this guy Carlos Abarca still working for TSB and if so, for how much longer? Another question: How many IT people does a bank with the size of TSB normally have?
At least his LinkedIn profile still lists him as “CIO and Head of Migration at TSB Bank”.
Maybe he meant “Emigration”.
mitigation
“You can use on of your eligible accounts to make a payment”
Top-class contracting. Fine reporting, Clive. But what is this “regulatory breach” of which you speak?
> CEO Paul Pester
“The aptronymic CEO Paul Pester.” Fixed it for ya.
Boilerplate FCA Standard — “Treating Customers Fairly”
I ran out of patience trying to document all the typos I found, screwy layout and formatting mess ups and other comparatively minor technical bodgery. There’s some weird Flash content (which most browsers block) on the login page, too. Goodness only knows what that’s about.
Being hacker-friendly? (the flash I mean, althoug by the looks of it, I’d say that if you wanted a honeypot hackers worldwide, you could do worse than what’s TSB doing)
It’s clear just from the screenshots you posted that they are pervasive. I was curious so I took a look at the login page myself and found more. It was also incredibly slow to load (as in several minutes) and looking at the network traffic I found a failed attempt to download a script from what looked like a test URL.
I’m also far from convinced that their validation is correct as it threw up some weird missing field errors for me on initial load before I even tried to submit anything (and for you, if the screenshots are an indication). The back button worked but seemed to take me back to a place that wasn’t quite the same as the one I left.
Well, It is certain that someone will be setting up a DNS-resolver for exactly that URL “fixing” the problem and leading to another, graver, one – Is it the people I know from former workplace relaxing on 4Chan, that likes to show “funny stuff” to people – or is it those “L33t Ruzzian Hax0rz”.
Beyond belief really!
You can bullshit the public
You can bullshit the management
You can bullshit the employees
You cannot bullshit the electrons.
Computers do not make mistakes, they execute yours correctly.
Time for the sixth phase of project management – punishment of the innocent.
These chips, they don’t lie.
Flash? Dear lord what were they thinking? I fully expect any commercial site I use is built with one overall goal – provide the service I need in the cleanest, most logical manner. Any eye-candy click-bait I can get on FaceBook, etc. No animations, no unnecessary graphics any more complicated than the company logo. Business Business Business!
> “clear information”…. “products that perform as firms have led them to expect”…
This is that famously dry British humor, right?
Alpha testing. That was my first thought when I started to follow this story. Probably what Pester meant to say was: The system is completely ready for Alpha testing and we will be giving clients a 10 pound credit to their account for each bug reported (knowing they wouldn’t be able to apply the credit because of the buggy software).
I was wondering if any back-up tapes that TSB had made before the changeover might be of any help at all as in being a base-line of where the accounts stood originally. It seems that the present database has been corrupted and perhaps irrevocably. All depends on what TSB’s back up polices are, I guess.
One thing that I have learned from this little episode. When I do my monthly full back-ups, I think that I will download the records of my own bank accounts. If, god forbid, they went belly up at least having my own personal records will give me a bit of leverage with my banks as they may be very well the only such records in existence.
Good job of reporting by the way Clive.
Typical IT backup policy is “three cycles”. A cycle might be a week, or a month. So if this mess continues for a longer time, the ability to incrementally restore will be lost at some point (three weeks or three months). I would suspect the actual time limit is four weeks, if that is their billing cycle.
If data is corrupted in transit, not in-situ, then the ability to restore depends on the transaction audit trail (X did Y at Z time). Not all data has such an audit trail, though one would hope the financial part is fully audited.
See vlade belwo. With banks, the amount of mission critical daily activity is orders of magnitude greater than in any other type of enterprise. In other organizations, most activity is stuff like e-mails and document generation, maybe some invoices and payments. By contrast, the overwhelming majority of records generated in a bank is about money going in and out. Screw that up and you are dead. Transactions happen way too fast for IT norms in other fields to be applicable.
backups would work till Friday. After that, you started to get transactions (such as Clive’s on Wed) through the new system. What are you going to do with them? Those are 3rd party transactions, and they want to be paid, or be sure they paid and don’t have to again (so what if the client didn’t get the money – they have a receit. It’s now between the bank and the client).
TBH, this can kill the bank – if a large number of transactions gone AWOL, how is anyone going to prove what should or should not be there? Give the system, if somoene hacks in, and empties it or messes up, what happens too? Etc. etc.
IMO, even if the system worked perfect from now on, it will be months for the fallout to be sorted out (and if the CEO thinks GBP10 + no overdraft fee will solve it, he’s insane), and I’d see some of the cases (as I’m sure some of it will end up in front of the court) dragging for years.
Whooah! Insanity unfortunately not;
What we are seeing is managements full and complete understanding of “The Trouble-shooting and Decision Making Process when working with Complex Systems” being put quite clearly onto public display here.
Don’t TSB have spin-people buffering the management stupidity so as not to spook investors and clients? Perhaps they left the impact zone already – or – they are all just wimmen (or gay) so their opinions don’t matter to “Mr Universe” here!?
PS:
The silver lining is that this is not a control problem involving a nuclear powerplant or a large petrochemical facility. It is only a Bank.
Well yes, it is “only” a bank, which is good in that potential mass death, or a power outage is not involved, but if the rent, mortgage, car loan, heating, or food bill is due, it is not “only,” it is f***ing catastrophic for the people involved. Perhaps because I am poor and not in the greatest health, but I keep seeing the trouble for the most vulnerable involved. If this was happening in the United States, you better believe some other institutions, be it the apartment owners, card companies, law enforcement, and so on, would see this as an opportunistic payday for them to extract more money from the victims, not as a chance to do the right thing.
But of course greed is good, right?
Silver lining dept.: judging by Clive’s account, a lot of people just had their credit card balances zeroed, and you have to wonder about the mortgages.
Debt jubilee for some. Maybe TSB can spin this as “maximizing shareholder value” and other banks will follow.
Or maybe not.
They’ve got IBM on the job now. Nothing to see here. All is well.
That really made me laugh. IBM are a shambling bunch of overpromising underdeliver’ers who somehow manage to pull the wool over the eyes of big corporations looking for IT “support”. This has the hallmarks of a talking point and being seen to be doing something rather than a serious attempt to get on top of the problems.
For a start, it can’t be emphasised enough, these are complex systems. Even getting a vague understanding takes several weeks. Coming up with recommendations and proposals for fixes and design changes that genuinely address the issues will take at least that, more like a month. The build and test will require longer still. There are no quick fixes to systemic and fundamental technical design faults or a database which has lost referential integrity (if it ever had it). Having people who are learning on the job will just make things worse. Unless of course TSB’s in-house IT department is merely a load of bums on seats and don’t know one end of an API from another. Which I wouldn’t entirely rule out…
Legacy cobol app Accenture ported to microfocus running x86 in two active DC”s , with teradata, infosphere and microstrategy on top. Oh and it looks like someone wanted some TIBCO too.
https://www.theregister.co.uk/2018/04/26/tsb_brings_in_ibm_to_help_resolve_performance_issues/
Let’s sprinkle in some cloud based AWS front end to fully leverage the tools capabilities too.
Yeah, it’s a little complicated.
Thanks for the laugh. I think you missed a few buzzwords, though. You should really be using “core competencies” instead of “capabilities”, for example. That was a favorite of the CEO of my last employer.
I used to work next door to the TIBCO building in Palo Alto, and one day I decided to try to figure out what the heck they did over there. I looked at their website but gave up trying to understand it after a few minutes. I just took another look at their site now, and apparently they are using a buzzword phrase generator that changes the phrase every 10 seconds or so. It’s as if they thought the parodies were cool and decided to imitate them. Bog help anyone who tries to use their “services”.
“Bums on Seats” … Reminds me of my Very First Job – In railway automation and safety systems. So, “we” need to commission this rather large train control system on a very specific date because lots of VIP’s need to see blinken-lichte and changing mimic screens for 1 hour.
Except there is a comms problem. The root cause being that the fiberoptic modules for the network are not delivered yet (This was hot and twitchy stuff in the 1980’s)
However, Everything is being done, All hands are on deck working Furiously beyond Capacity. Meaning –
We are sitting with blinking network analysers and logic analysers in about every node along the track, to fit everyone in (obviously there is a system management system that one can ask about everything, but this is not “visible” enough), 3-5 people in each little shed, running overtime, reading, having food delivered, absolutely racking up “Activity Metrics” – while some factory in Germany is producing, at its very own pace, the fiber optics modules. All this because The Minister will not tolerate, absolutely not, a 4 month delay in the commissioning. So, we commission.
It was in the summer, so it was very nice and cozy; Only mishap is some of my colleagues got chewed over for being caught with a barbecue by a journalist. Fun and Profitable times.
Well, really, what is the problem? At not one, not two, but three old jobs of mine they just faked the screens for the VIP presentations. Anyone remember that old DOS program, bogosity?
All three co’s were trying to turn their biz experience into a salable IT product. One was a successful small hardware specialty mfgr, another a very successful specialized equipment rental outfit, two were family-owned and very good in their own fields, the third guy just had the company fall on him (long story, not rerlevant here) and got involved with some dodgy partners when it looked like his biz, a video ancilliary, was going obsolete. Unfortunately, the brass in all of these companies were totally snowed by their IT guys. In the first two cases I put it down to their not being used to hearing such stunningly pure BS, which was close to 200 proof — their businesses mostly involved real people and real things and if those things didn’t work you could tell pretty quick. The third guy was — well, I believe that great philosopher W C Fields observed that you can’t cheat an honest man.
I was in accounting for all three co’s and we lower-level creatures knew, but Management (genuflects) were all keen on the meetings and the lunches and the $$$$$ (wow!) and you just couldn’t tell them anything.
It was highly educational for the mgmt of all cos. The two family and well-intentioned ones survived, the dodgy one did not.
Not likely to produce a quality fix within any reasonable time frame with a hyper-masculine heavy blamer like Pester leading the team. He has already wildly over-promised. His IT, in or outside, will see a lot of turnover and reinventing of the wheel. The first person who should go in this debacle is Pester.
“Whaddaya mean, I can’t pull my SUV out of that ditch with these kittens? I’ve got a whip, haven’t I?”
Clive
IBM is on the job. That’s been a laugh for a lot of people for 50 years. (I won’t go into their war record here.) A local friend, former IBM Canadian national sales manager during the introduction of the Selectric, wrote in his locally-distributed autobiography about how many, if not most of them, failed after much hype in their introduction. The sales and service people were “persuaded” to tell customers that such failures were: conversion errors on the part of users, or rare faults probably the result of damage in shipping. They were “persuaded” to stall customers because replacements were not available, or were having the same “unusual” problems. In other words, they were forced to lie like old rugs in order to retain their jobs.
He quit IBM shortly thereafter.
Thanks for your public service announcement, Clive.:-) And to think I was considering moving my account to TSB. I delayed and, lucky for me, I did.
Rev Kev
When you state ” belly up “, do you mean that there is potential for the above to occur within other banks ? I have difficulty understanding all of the technical side, but have read enough on this site to know that it appears that dodgy IT is widespread within banks. Might it also be wise to keep an amount of cash on one side in case of such an eventuality ? – heaven knows what an incident like this would be like in a demonitised society.
Great work Clive.
What I mean to say is this could be potentially possible in a lot of other banks. From what I have read here, the IT departments of banks are not getting the support that they should be getting or even the respect. Even now there is a Royal Commission delving into the behaviour of banks in Australia and some of the stuff coming out of it is actual criminal shenanigans so god knows what the IT state of these same banks are. It’s not gunna kill me to back up my banks records once a month and having some cash in reserves is not a bad idea either. Why assume that all is good? As they say, when you assume, you make an ASS out of U and ME.
Having seen what goes on behind the scenes and how widespread these problems are in the entire industry I would say they are pervasive no matter which bank you’ve dealing with.
Put it this way, I do not rely on a single institution for access to funds and the ability to make payments. Ironically, TSB was one of my “back ups”. Not wishing to get all prepper-minded, I now keep £100 cash on my person at all times and a few hundred at the house — enough to survive for a week or so on. Hopefully I’ll be proved wrong and never need to rely on my off-grid (~ish) stash. But nevertheless I have one.
This is beginning to sound like an old 50s’ ‘Film Noir.’
“Make that small bills. Used. And nobody do anything stupid and no one gets hurt!”
The big worry here is about something like this happening with the State Services programs. What if the American Social Security payments system were to crash? What bank, or vendor will extend credit to those affected while the ‘outage’ is worked through? We’re into imaginary numbers, in the literal sense, here.
Anyone talking yet about nationalizing TSB?
Are you kidding? We only just (in effect) de-nationalized Lloyds after it had to be rescued during the financial crisis, courtesy of its rescue of HBOS — that’s why TSB had to be split off from it in the first place: too few big banks were left to satisfy even the UK’s placid regulators.
Money in your mattress? Gold coins?
Thank you both.
Hopefully I might be able to improve on my coin bank :)
It is not as if we live in an age of certainty in any form.
I feel sorry for all the code-monkeys who are working 14-hour days trying to fix this shitshow. The worst part is that it’s a retail institution so highly mission-critical and visible. And it could very well be the demise of the bank.
But this type of situation is not really that uncommon in systems conversions. The first such conversion I did, for a small manufacturing company in the late eighties, had to be rushed because the original system had been programmed with a hard end-date by its long-haired programmer in the seventies – just prior to his driving off into the sunset in his VW. A rushed conversion resulted in bad ledger data for three months. But – business life went on – customers were billed, bills got paid, inventories managed – but the system produced no reliable general ledger data for months. Inventory of negative $80 million, for example. So for this period , I made up the company’s monthly results based on the actual sales and budgeted or near-budgeted figures for the rest. No one at the home office was the wiser and we had time to get the thing all fixed up by the time the auditors came in. Whew!
The poor beggars in the TSB’s IT dept. are absolutely naked and exposed and under the gun every minute. And once the thing is done, they know they probably won’t have jobs or many prospects considering where they work now. Just a shitshow deluxe for all involved.
LOL! Ours was hiking in the Andes, or so we were told to say.
In commercial retail, usually the inventory system was the last one developed, because of course it was more important to develop and install accounting software to reduce costs, and sales tracking software to screw the sales people. In the 1970s the small wholesale office supplies company I worked for contracted for an inventory control system. Among the items were X-acto knives and accessories, including a package of assorted router blades. When the company submitted the list of products for review, one was an ‘assrouter’. Our owner/manager commented, “I didn’t know we were in the medical supplies business.”
Canada’s federal civil servants, as well as many pensioners, are living (in hell) through the government’s conversion into the Phoenix payroll system, being overpaid, underpaid, not paid at all, not getting pensions, and a host of other problems. The system was contracted by the former Conservative government of Stephen Harper, which bunch knew it was prepped to fail, but rushed into implementation by the succeeding Trudeau Liberals. So, you see, in the white north, we too have Conberals, or maybe Libervatives. Neither gives a family blog about us mopes.
I have a TSB mortgage and have tried to login this week. My mortgage was originally with Lloyds but was bundled into TSB during the split. We are stuck at TSB because our pre-crisis mortgage was commuted to permenant 0.5% above base rate tracker by the govenment during the recession, without any publicity or notification i would add, but obviously now it’s permenantly cheap and we would be foolish to move it elsewhere.
Until today (friday) I couldn’t login at all. Today i finally successfully logged in, but received an unexpected message:
‘You can’t access your accounts online We’re sorry, but you don’t have any accounts that can be accessed through Internet Banking. Please contact us on 0345 975 8758 for further assistance. We’re open 24/7.’
Which is odd because I have a username, password and login details that the system clearly recognised, ie this wasn’t a login failure.
To those who ask what may happen if there’s a natural disaster, I have this little tale to share:
On August 29, 2005, Hurricane Katrina came stomping ashore along the Gulf Coast of the United States. I’m sure you’re well aware of what happened next, so I won’t elaborate.
Well, the Gulf Coast has financial institutions, just like the rest of our country. How did they fare?
I can’t speak for all of them, but I do know a lady who worked at Merchants & Marine Bank, which is headquartered in Pascagoula, MS. The bank came through the storm with flying colors. According to my friend, when they brought their IT systems back up, every morsel of data was still there.
So, if an itty-bitty bank in southern Mississippi can come through one of the worst hurricanes in American history without losing any data, then TSB doesn’t have any excuses for poor IT.
Great report. The illustration “LimitManagement” window is interesting (if that’s the right word). Asks you to do data entry while rejecting any valid zero sums. There’s something wrong there.
Your credit card is treated as new with no transaction history, although it’s a 2 year old card….with no transaction history. Are the system parts not talking to each other, or was the data not imported, or was the data imported wrongly, or were all the file and transaction create/modify/access timestamps fubarred on import or never imported, or, or, or…..
This TSB IT migration fiasco is much much worse than I first thought.
Yes, if I had to guess, there was a TSB management decision to ship the new system accepting all “cosmetic” or non-functional defects. That’s because there’s simply so many of them it must have been intentional.
That then crept into accepting all non-critical defects like mangling account history data. The thought pattern became hit the deadline date regardless of the cost.
From that point on, it was mere footsteps to “just throw in any old rubbish, quality is simply not a deal-breaker”.
And now here they are.
Paul Pester is the perfect man to keep this situation in the news.
Think it’s a shitshow now?
Paul is going to supersize it, with fries!
This reminds me of the botched Obamacare roll out that cost taxpayers hundreds of millions of dollars. Whose going to pick up the tab for TSB?
The cynic in me says that Lloyds will buy the pieces for a song and take a year or two to integrate the worthwhile bits and ‘disappear’ the rest.
Unless the government steps in, the main losers here will be the customers. Consider it a giant ‘buy in.’
I’d also worry about the viability of the Spanish ‘parent’ bank, Banco Sabadell. How much of the cost related to this is the parent bank responsible for? If all of it, and TSB is 25% of the new bigger bank, then, roughly, Sabadell could end up losing a quarter to a third of their original ‘value.’
In the spirit of Brexit could the UK government not just blame the spanish for everything and rejoice in a scandal that is not of their own making?
So I’m a long time IT guy. CIO of a fairly large county in Florida, ex-EDS guy. But never done banking. I would think that, because banks have a high transaction rate, are 24×7, and worst of all, the transactions must be ordered, that you would have to start any programming effort like this by doing a real-time database replication.
So if I have some stodgy old C-ISAM or DB2 database on a mainframe, I would set up a shiny new Oracle or SQL Server database on my fast new storage system, and replicate all new data to both databases instantly. That way I could write new code against the new database, without screwing up the old one. How else could you do it and be able to test?
Is there an expectation that, with so many moving parts, TSB’s or IBM’s programmers can figure out what wrong things are happening while the plane is flying? What TSB has done seems to amount to criminal negligence.
Appreciate the comment and insight.
My take is that when you completely outsource a function such as IT (or have it all done by a now missing parent), the people left inside – especially those who have been discarded in a divestiture – no longer know how to supervise, direct and task those on the outside. Resilience is deemed an attribute too costly to maintain.
Outside contractors – unlike an in-house IT team – are driven by their managers to maximize revenue, not product effectiveness. If they can lay off liability onto the client, say, through a one-sided service agreement, every fix becomes cost-plus. The arrangement can quickly degenerate and finger-pointing ensues. Pretty much what TSB looks like now.
TSB was relying on its corporate parent for its IT. Its parent had no interest in devoting time and resources on a unit it was selling off. It just wanted the money and to be contractually off the hook. The Spanish buyer doesn’t seem too bright or too knowledgeable about a core competence in banking. This will not end well or quickly. Customers should flee.
IF they can get their money out, which seems to be a problem.
I was wondering about more or less creating a sort of parallel that would allow the system to develop without interfering with the existing, almost like a redundancy.
An FT article said the process to transition started in 2014 and that the annual cost to continue using Lloyds was something like $100 million a year.
” But it has no transaction history, sees it as a new card when it isn’t.”
Does that mean he doesn’t have to pay the previous bill? This could work out – for Clive. Does he know yet whether merchants are getting paid when he uses the card? If so, he might want to use it a lot. :)
I THINK that’s a joke. But it could be a very expensive one for the bank.
” So while payments are “working” on the TSB system, as in, not being declined, they aren’t working end-to-end in terms of posting to the customers’ accounts from the back-end through to the front end account servicing layer.”
Free money, Clive? Risky, but I’ll bet some people will see it that way.
Seems there’s a “technological issue” with the Toronto Stock Exchange.
Hmmm…might we have gotten ahead of ourselves, automating everything? :)
Hope I’m around when the over-complicated over-connectedness finally burns out…
Thanks for the post. I just tried to get into the website, 3 different pages:
top page http://www.tsb.co.uk gives
Internal Server Error – Read
The server encountered an internal error or misconfiguration and was unable to complete your request.
Reference #3.120ad817.1524861564.5d601b2
so I thought I’d contact them to let them know, and for page: http://www.tsb.co.uk/contact-us I get:
Internal Server Error – Read
The server encountered an internal error or misconfiguration and was unable to complete your request.
Reference #3.120ad817.1524861660.5d8dae3
Ho Hum ! Did I also read that they’ve hired a team from IBM to help them fix things ? Now its gonna be REALLY screwed up.
From the TSB website:
Oh really!
Including the Current Problems with the IT system?
The good words don’t match the image of Mr Pester mentioned above. I don’t see how ” bragged about competing in endurance sports such as triathlons to demean less physically fit colleagues), Pester is suggestive of a hypermasculinity personality disorder,” fits with the values expressed.
Surely Mr Pest is a kind, considerate, loving boss, looking out for every employee’s and contractors welfare. The corporate values are clearly expressed.
Clearly TSB has been running little tests of their systems, all designed to give the boos’ confort that Nothing Can Go Wrong…..Go Wrong…..Go Wrong
Little tests: Pester’s Testicles.
Reminds one of the British Airways system outage a few months ago. Similar reassurances by the brass, similar complete disaster behind the scenes that was obvious to all.
They took the concept of Agile Development too far. When the proponents of Agile mention TDD i.e. Test Driven Development, they don’t mean it’s the customers doing the testing.
They thought they were doing TTD – Test To Destruction.
Apparently, the platform was developed in-house by Sabadell, using its own IT division, Sabis. The new system is a reportedly costing TSB “hundreds of millions” but this could well end up in the courts. It also brings into question the relationship between a parent company and stand-alone subsidiaries with different regulators.
A different take on the fiasco from ‘This is Money’ which raises the questions on adequacy of regulatory capital for Sabadell, which has been on a bit of an acquisition spree for the last ten years.
http://www.thisismoney.co.uk/money/article-5669453/TSB-heading-legal-showdown-Sabadell.html
Whatever else (and I am not an IT or banking person) common sense tells you that a new system should have been thoroughly road-tested before going live.
The ‘date’ issue picked up above is not an isolated instance – I am trying to obtain delivery of a package which is with SEUR, a Spanish courier business. No email address of course but fill out the online form. Except it has some anomalies in it which prevent entering a non-Spanish phone number and it also has a glitch in the zip code/City match. Written by IT people who don’t live in the real world.