Based on reading the Guardian’s live blog on the Parliamentary hearings on the TSB IT meltdown, plus some additional coverage, no one appears to have comported themselves all that well, but CEO Paul Pester put in such a spectacularly arrogant and disconnected-from-reality performance so as to have somewhat diverted attention from the poor job MPs did of questioning him and other TSB and Sabadell execs.
Even though the personal styles were as far apart as one could possibly envision, there was an eerie similarity between Pester’s fabulous disconnect from reality and that of former Wells Fargo CEO John Stumpf. Recall that Stumpf ran an operationally sound and highly profitable bank that was being discovered to have a sales culture that could only be characterized as criminal. As Elizabeth Warren pointed out, a teller stealing a $20 from a drawer would go to jail, yet Wells was engaged in the same sort of thing at an institutional level. Yet Stumpf blandly kept insisting that this was just a few bad apples in the face of red hot rejection by Senators and Congressmen from both sides of the aisle.
It took two rounds of disastrous Washington appearances, plus the press picking apart Stumpf’s dodgy claims before the board roused itself to tell Stumpf he needed to resign. It was not hard fto show that the head of retail banking was driving the sales process and punishing any managers who questioned it, and Stumpf kept setting ever-more aggressive targets. But Stumpf held on far longer than I had imagined possible. That was likely due to misguided personal loyalty, compounded by them mistakenly believing that his reassurances were based on better intel than what board members could read in the media.
As much as the Parliamentary hearings made clear that Pester is utterly clueless about what is going on with TSB’s systems and a PR liability for the bank, he’s likely to hang on for an even more unseemly amount of time. First, his bosses at Sabadell seem just as out to lunch, so they are unlikely to see him as a problem, since that means admitting they screwed up disastrously too. Second, who could possibly want to step into this role? Any insider is presumably part of the problem. Who would be willing to take on a bank that could have fundamentally unfixable IT problems, which means having to resolve it in a novel manner? Normally banks go tits up in familiar ways, like stupid loans and embezzlement, as opposed to an institution of scale making such a mess of its records that it dies under the liabilities.
A few points from the hearings:
Lame questioning by MPs. It was shocking that no one asked even a slightly technical question. As our readers have pointed out, even by looking at Twitter screenshots and connecting the dots from press reports, you can infer quite a lot that is damning on the IT front. Our Clive pointed out one shocking planning failure that should have been a lay-up as a line of inquiry:
When I read the communications from TSB, there were a couple of stand-out facts which had me shaking my head. The first was that TSB were moving their entire customer base over to the new system in one go. This required a two day (the weekend of 21-22 April) complete system outage which implied it was only just about feasible to do the transfer and the data load in that timescale. This meant that there was no possibility of pre-live testing. And that there would be a single Critical Success Factor to the migration which was simply measured as getting all the data over to the new TSB system and the imports completing. Whether that data was the right data for the fields being populated in the TSB system and whether fields were mandatory or optional in terms of getting data inputted into them before the start of the on-line day on Monday 23rd was never apparently considered. There are now big gaps in the historic data on the TSB platform – I’ll cover this more below.
And on Tuesday, before the Parliamentary hearings, a Wired story volunteered the notion that TSB had tested its systems over the migration weekend. Um, by the time you’ve migrated your production system, you are past the point of testing. Vlade poured cole water on this idea:
TSB testing on the weekend. Experts, huh?
You DON’T do testing of migration on the production system at the weekend of the migration. That would be even more idiotic than not doing it at all TBH, as it has all the potential to screw up the production, reduces time available for migration and any inevitable cock-ups (and hence also ability to roll back), and a lot of other problems I could think of.
The testing of migration is done on parallel test systems (that are, as far as practicable) perfect copies of production (some brave souls could use the DR site, which of course is ok until a problem strikes and you need that..) – so called dry-runs. You may do those on the weekends (to simulate the whole migration, so for example be able to detach/re-attach integrated systems etc., plus it takes a lot more of effort than usual working day), but not on the weekend of migration.
It’s expensive to have a full parallel system, which is why partial migrations are preferable. But to migrate a full system like TSBs w/o at least two full-population dry-runs (first one tends to find quite a few problems, second hopefully fixes them and establishes better runbook) is negligency par excellence.
Of course, then you also need to have a way of testing the dry-run – otherwise I can dry-run trivially (>/dev/null will always work… ) and claim sucess. In my experience, you might have done a dry-run on the weekend, and then spend the rest of the week testing and evaluating it.
There has been more than enough evidence via Twitterverse of other IT horrors, like errors in multiple programming languages (confirmation of the fear that the problems are in many sub-systems), internal testing domains on production, tons of internal error messages visible to customers, and shockingly incomplete customer interfaces when they do work (numerous typos and formatting errors). As Richard Smith said, this looked as if it had been planned and executed by high school kids.
As a result, MPs were not able to do more than mildly dent howler assertions by Pester. The MPs should have, between access to IT experts and a better view of at least some of the issues from constituents, should have been able to bore in. Instead the level of questioning was based on grim anecdotes, with Pester able to assert that he had a better view of the situation, and kept repeating the sort of blather that had been getting pushback on Twitter for days, like his assertion that his data showed that almost all customers were getting online.
Nicky Morgan did well despite being disadvantaged by not introducing a technical take. She had two colleagues with TSB accounts try logging in during Pester’s testimony. Neither could get access.
And this exchange, as recounted in the Guardian, was mind-boggling: