I have no idea whether this estimate is still valid, but back in the days when I was working with O’Connor & Associates, the head of technology (and O’Connor had a well run technology operation) said the cost of documenting software (as in having the developers write up what they did in sufficient detail so a third party could maintain the it) was an additional 20%. It was very difficult to get traders to pay for it, ironically not so much for the hard dollar costs, but the fact that IT was resource constrained. The idea that the coders would spend a lot of time on leaving a paper trail, rather than getting to a business center’s urgent need, was a hard sell.
But Lehman illustrates what happens when you do what most places do, and skip documentation. Now of course, modern IT on the Street was supposed to be using more off the shelf software and doing less custom development, but you would never know that reading this Financial Times story:
Unwinding derivatives is a complex task at the best of times. In the case of Lehman, one of the biggest dealers in some of the most complex derivatives markets, this has been even more so. Lehman’s global derivatives book included contracts with a notional face value of $39,000bn and deals with 8,000 different counterparties when it went bust. The derivatives business was actually split into multiple strands, backed up by between 20 and 30 different systems.
Once it went bankrupt, the staff who supported these systems “evaporated”….
“The more time goes by, the less insight remains in terms of the people who staffed those systems,” said Mr O’Hanlon. He likens the amount learnt about derivatives from the Lehman job to lessons from wars. “The medical industry has advanced itself whenever there is a war, when it has to undertake hundreds and thousands of procedures in a compressed period of time.”
The main conclusion so far – and there are many fascinating details in the Lehman unwind – is that the technology costs for derivatives have been underestimated. In addition, the answers that every regulator now wants and every investor should demand, in terms of levels of exposures and risks that banks hold, cannot be easily gleaned from the current multiple systems used to value and track positions.
“Many previously hidden costs of running a derivatives business, including technology support of multiple disjointed systems, can no longer be discounted,” says Luc Faucheux, a managing director at Lehman Brothers Holding Inc in New York.
34 comments
Comments are closed.
personal observation: broker/dealers stuffed derivatives trades into existing fixed income trading systems and some rolled out dedicated systems only in 2006, others had plans for later dates. reporting and anything but 10,000ft look at risk management on the entire book were extremely difficult. this does not mean one could not get exposure by name, but it was only possible through ad hoc queries.
Yves,
There is nothing like a bankruptcy if you want to perform a post mortem. The obvious answer to me is to point to poor regulation and need for reform but those who argue that the core of the problem was that Lehman was allowed to fail probably feel vindicated.
The problem with trading software tends to be split multiple ways:
– The off-the-shelf systems need configuration, often quite a lot of configuration. You’re not buying Microsoft Office when you buy one of the bigger packages, but a set of building blocks that then get customised for the specific business area’s needs. It does save a lot of coding but you still end up with something semi-custom. While the name tag on the system might be familiar, two instances of the systems might still be rather different in their configuration. Once you factor in the non-existing documentation (because this is investment banking and everything needs to be finished the previous week) and the institutional memory that left the building in a hurry when a company goes bankrupt and you’re left with someone else’s reverse engineering opportunity.
– There are still a lot of custom systems around in specific business areas, usually in the less vanilla/more interesting areas. Which, as we all know, are the ones that carry the least risk of blowing up, right? These systems have often been around for a decade or longer and have gone through several complete changes of staff working on them. Again, this can cause problems with the institutional memory (and often does) and as these systems have grown over such a long time there tend to be plenty of areas that aren’t understood very well. Documentation? Hey, Trader X needs this feature by last week so he can get three trades off an Excel spreadsheet.
– IT departments do tend to have quite a high turnover for reasons that aren’t really bank specific. Unfortunately it is still the case that in order to climb the ladder, IT staff and especially software developers often have to change companies if they don’t want to get stuck in a particular role for the rest of their conceivable career. This again is not an investment banking issue but more of a general issue and again it doesn’t help with the institutional memory in absence of documentation.
None of this is specific to Lehman but more of an issue with the overall way that IT systems tend to be developed, maintained and what the customer’s priorities are. Add to that the fact that a lot of trading systems development consists of chasing and integrating the latest deal types or models that the traders are developing and not documenting the existing ones. The fact that the remains of Lehman now have an issue with those systems doesn’t surprise me.
Disclaimer: I’ve worked on various trading systems for a variety of banks both as a developer and a team lead so the comments reflect my experience. I haven’t worked for Lehman but given that this specialised field tends to be rather incestuous, I’m seeing the same issues over and over so I doubt that Lehman was different in any way.
COTS isn’t really all it’s cracked up to be … you just end up with another set of undocumented bugs to deal with for example, and a whole new knowledge set you need before you can even start. The only ‘plus’ is that this is handled by a third party you can ask later on down the track. But that isn’t much of a help when the logic is coded in some proprietary system glued messily to another, ad infinitum.
The other issue is any complex software is simply complex – no framework or tool can remove the complexity. Although coders might joke that ‘if it’s hard to write it should be hard to read’; this is simply a reality of any complex system. It’s simply the outcome of the fact that almost every non-trivial piece of software really constitutes some of the most complex machines ever designed by humans. It’s more like an oil refinery than say a car or truck.
Oh, and documentation takes too long to write and even longer to keep up to date. You’re almost certainly better off with no documentation beyond the source code, vs documentation which doesn’t reflect it – which is more often the case when it does exist.
I wonder if the knowledge has ‘evapourated’ on purpose anyway.
(i’m a software engineer, but never worked in a bank)
Concur with above statements. Take your car to too many mechanics and the last person to touch it has to spend more time figuring out what was done before than the fix it self.
Hot-roding off the self systems is a cheap and messy, admin created nightmare, shez some people always try to look good going the cheap route and when it fails blame the coders, although I do know many that have made small fortunes in this niche, after the fact.
Disclaimer only C++ for fun now with valve engine mods after playing liaison between software and admin conflicts once to often.
Skippy…Between you and me, Lowry, this – no no! department – tell records to get stuffed!- is about to be upgraded.
For you Lone C++ Coder, Enjoy:
http://www.youtube.com/watch?v=LFlFIG22Y9E&feature=related
there are so many issues here it would be imposible to sum them all up here. what you have to know is that at any one time NO ONE EVER KNOWS how it all works. the systems which enable the global financial system are complex beasts and to understand even a single system truly you need not only someone very highly trained and motivated but someone who understands the full context of a system the traders who use it the business which is done through it. the algorithms and models which operate on it. the inputs it takes from markets and the numbers which come out of it and what they mean. In the Lehman case the firm had gone throgh 3 waves of redundancies pre bankrupcy where the human and intellectual capital of the people who ran many of those systems was eroded like a tsunami. when the non IT literate administators turn up they can barely operate a spreadsheet let alone a Complex exotics Off balance sheet fixed income trading and valuation system. because supporting and developing such systems is expensive and requires 3l33T 34NK1N $kills. and this is before the corrupt management and traders who fiddle the trade books for the benefit of PnL… and stuff… its just how it works…
It is not only the technical documentation that is lacking. Very often the customer cannot be bothered to write up a decent functional specification describing what the program is supposed to do. Most of the time a new program ‘has to do exactly what the old program did, minus the obvious problems with the old program’. Later on it gets even better, because when new features are introduced in an existing program functional documentation does not get updated at all.
And so you end up with one semi-retired bearded IT guy who never left the company because that idea just didn’t come up in his head who remembers what the very first program did and why it had been developed. If he can be bothered, he’ll explain the purpose of the program, but he cannot read the new object-oriented programming language.
oh sure, blame it on their software the fact that they have poor documentation ON THEIR TRADES. anyone who worked in a bank can see this for what it is: blame a business problem on IT / cover your ass. it’s PR aimed toward getting upper management to divert measly IT bonus money to trading staff. yves, the fact that your blog is swallowing this nonsense whole is really really surprising.
The theologian Reinhold Niebuhr, in The Irony of American History makes a similar observation:
We have forgotten to what degree the wealth of our natural resources and the fortuitous circumstances that we conquered a continent just when the advancement of technics made it possible to organize that continent into a single political and economic unit, lay at the foundation of our prosperity.
And yet the myth that it was our superior diligence, our greater skill, our fervent devotion to the ideas of freedom and capitalism persists. As Arendt goes on to explain:
Wealth and economic well-being, we have asserted, are the fruits of freedom, while we should have been the first to know that this kind of ‘happiness’ was the blessing of America prior to the Revolution, and that its cause was natural abundance under ‘mild government’, and neither political freedom nor the unchained, unbridled ‘private initiative’ of capitalism, which in the absence of natural wealth has led everywhere to unhappiness and mass poverty.
I think the only explanation as to why these blatantly false beliefs have persisted is that, in the era of neo-colonialism, they served the interests of many Americans, and not just the rich. As George Orwell wrote in Rudyard Kipling: “We all live by robbing Asiatic coolie, and those of us who are ‘enlightened’ all maintain that those coolies ought to be set free: but our standard of living, and hence our ‘enlightenment,’ demands that the robbery shall continue.”
From a purely materialistic viewpoint, I have identified five entities upon which our wealth and economic well-being depend:
•Land
•Labor
•Capital
•Natural Resources
•Science and Technology
Land had its great champion in the Catholic Church (the feudal system).
Capital had its great champion in Adam Smith.
Labor had its great champion in Marx.
Science and Technology (or at least its practitioners), however, never seem to have had their great champion.
And of course the value of natural resources and foreign labor are always marked down or outright dismissed in a colonial or neo-colonial regime (like the United States).
it will take some time but is not insurmountable. The talent is out there for the right price and sometimes there is no right price for eating it. If individual positions can be enumerated and sliced and diced, then the derivatives can be remic-ed into something new which may require less time than attempting to learn how the original system worked.
What I recall from bankruptcy proceedings is that netting does not occur in the institution and everything settles bilaterally. So what ever is in the system is really a different product in bankruptcy because those long chains of netting no longer apply.
The parties that won will be coming forward, the parties that lost will be keeping quiet. In this respect the lehman bankruptcy will be costing far more than it should. If lehman can’t dig up a particular losing leg some bets previously thought to have gone south may actually become shining gems in the portfolio. Counter parties aren’t going to help U out too much here.
This is all hypothetical speculation, but it makes sense to me.
«the cost of documenting software (as in having the developers write up what they did in sufficient detail so a third party could maintain the it) was an additional 20%»
That is very optimistic if that refers to development costs. It is about reasonable if the base is total IT spending.
A famous author (Brooks) said that if writing some sw costs 1, turning it into a maintainable product costs 3, and if it is system/complicated sw that involves another factor of 3.
That is BTW one of the 2 reasons that free software is an economically sustainable activity: because just *writing* software is rather cheap. Turning it into a product is far more expensive, and selling and marketing it, and profits for shareholders, are even more expensive (most companies that sell software spend less than 15% of sales on developing the software; salesmen/marketing expense accounts in many cases total more than the cost of developing the software).
I have worked on the Street and elsewhere in IT and very rarely is IT given a whole lot of respect – let alone listened to when they dare make recommendations of needed actions that would incur cost. Disrespect your IT staff, ignore them, punish them for telling you the truth and this is what you get.
I think I speak for every person in IT when I say:
d’oh!
Touché
«the cost of documenting software (as in having the developers write up what they did in sufficient detail so a third party could maintain the it) was an additional 20%»
That is very optimistic if that refers to development costs. It is about reasonable if the base is total IT spending.
A famous author (Brooks) said that if writing some sw costs 1, turning it into a maintainable product costs 3, and if it is system/complicated sw that involves another factor of 3.
That is BTW one of the 2 reasons that free software is an economically sustainable activity: because just *writing* software is rather cheap. Turning it into a product is far more expensive, and selling and marketing it, and profits for shareholders, are even more expensive (most companies that sell software spend less than 15% of sales on developing the software; salesmen/marketing expense accounts in many cases total more than the cost of developing the software).
«the technology costs for derivatives have been underestimated.»
This is just a general case of a principle that I (re)discovered: that essentially all financial fraud (as opposed to simple embezzlement) is based on under-depreciation, that is running down assets. eating the seedcorn.
It is the age-old sharecropper problem, the quintessential agent-principal issue: that anybody who is given temporary control over an asset and gets a share of profits, will try to boost profits by running down the assets with too little depreciation (maintenance, repair, sustainable levels of use). Recognizing early the income and late the expenses associated with deals does the same of course. “Picking pennies in from of steamrollers”, ignoring longer term risks and failing to provide for them, is another variant.
The reason why this is almost always the basis of any fraud is that it is difficult to prove that under-depreciation is happening, and then things blow up because of it, then usually it is too late.
The lack of software documentation is system wide. I doubt that there is a bank in the world that can say that it’s trading desk software is adequately documented.
Now as to winding down transactions, that begins with the individual contract and the amendments to it. That would include side letters. Most troublesome will be the side letters, those are probably not recorded in the contract file.
To assert that the difficulty in winding down is that IT people have left is a gross mis-direction. The difficulty in winding down is that the people who know which agreements/amendments have not been recorded is the problem. For that there is an expedient cure, in the absence of substantial proof by the counterparty, repudiate all contracts.
More likely is the fact that in the process of winding down, the cost of that action is far greater than anyone anticipated.
q,
There is undoubtedly a great deal of truth to what you say.
It all harkens back to something Keynes wrote to his lover, Lytton Strachey. “I want ot manage a railway or organize a Trust or at least swindle the investing public,” he said, “it is so easy and fascinating to master the principles of these things.”
So while some top intellectual talent has undoubtedly gravitated to Wall Street over the past couple of decades, that gravitation pull certainly hasn’t been to the trading pit or executive suite. And like you say, the real brains behind the operation certainly don’t bring down the kind of pay the thieves running the organization do.
That’s why pseudo-scientific nonsense like this from Luigi Zingales (see yesterday’s post “Capitalism After the Crisis” and “Free Markets” Newspeak) is so grotesque, and would be risible if it weren’t for the fact that this type of claptrap is used to justify the exorbitant salaries of traders and executives:
The explosion of wages and profits in finance also naturally attracted the best talents–with implications that extended beyond the financial sector, and deep into government. Thirty years ago, the brightest undergraduates were going into science, technology, law, and business; for the last 20 years, they have gone to finance.
Shoo! One wonders how Zingales keeps a straight face when he says these kinds of things.
But I think there’s a much larger question here, one that Hannah Arendt alludes to in Crises of the Republic:
The enormous growth of productivity in the modern world was by no means due to an increase in the workers’ productivity, but exclusively the development of technology, and this depended neither on the working class nor on the bourgeoisie (capitalists), but on the scientists. The “intellectuals,” much despised by Sorel and Pareto, suddenly ceased to be a marginal social group and emerged as a new elite, whose work, having changed the conditions of human life almost beyond recognition in a few decades, has remained essential for the functioning of society. There are many reasons why this new group has not, or not yet, developed into a power elite, but there is indeed every reason to believe with Daniel Bell that “not only the best talents, but eventually the entire complex of social prestige and social status, will be rooted in the intellectual and scientific communities.”
(It should be added that Arendt did not include the field of economics, nor any other of the social disciplines, under the rubric of science. Quite the contrary, she was most adamant in insisting that the behavioral sciences were a very different animal than the natural sciences.)
Blissex, very nicely put.
That is the scam all right: one or ideally both of a) goose earnings unsustainably b)cut costs unsustainably. Then take your money, paid out of short term profits, and run. The Lehman IT situation is a toy version of the whole Lehman balance sheet.
q, read more carefully. Neither Yves nor the FT is pointing the finger at anyone in particular. Identifying a culprit is actually more difficult than either you or the FT’s source would like: which is precisely what makes the scam so attractive.
The above comments (plus original post) represent a most interesting contribution to non-finance, non-techie people such as me. Appreciated . . .
Until management in any field gets it through their heads that IT isn’t just “The help,” and that software can’t be treated like staplers, copiers, or other office equipment, this situation will repeat and continue.
Unfortunately, most of the newly minted ignoramus/MBAs I’ve met seem doomed to this repetition.
I disagree. Whoever absorbed Lehman (JP Morgan Chase?) should have locked in the appropriate Lehman staff who could help with the conversion work. But, I do agree that there is little (no) written documentation. The priority for programming resources is developing more apps, and besides few programmers have any talent for writing clear documentation.
The Lehman bankruptcy was messier than that. This is all off the cuff and may not be right in every particular so if you are better informed please be gentle with me (and give us all the correct dirt!).
Nomura got the equity and I think debt trading pieces of the UK end of Lehman. They were *very* quick to lock in the IT staff they needed to support those systems, and in fact were so welcoming to the Lehman guys that they caused a certain amount of FUD amongst their own pre-Lehman IT people. Lehman UK would also have needed recapitalisation since Lehman was in the habit of repatriating all the trading capital to the US as weekends. All the cash was in the States when Lehman went TU. So: quite a commitment by Nomura there. They will be careful of their investment, as best they can.
BarCap got the U.S. equity and debt trading. I don’t know whether they were as energetic as Nomura in grabbing the IT staff they needed but they are reasonably switched on so their is a fair chance that they did lock people down as much as possible.
Lehaman’s private asset management business, Neuberger Berman was sold to its management at the end of ’08. The customer assets in there would almost certainly have been held separately from the Lehman Bros stuff, so there wouldn’t necessarily be a huge mess to resolve. One sort-of exception might be any Lehman structured products sold to Neuberger customers, which products will now be worth cents at best. That will make Neuberger’s clients irritable. On the systems side there wouldn’t necessarily be much more than reporting links between that part of the company and the rest of Lehman (maybe order routing type stuff too), so you would expect the systems to be reasonably decoupled.
Which leaves two big Lehman Bros messes – Prime Brokerage, mostly in the UK, and the global derivatives business – and probably a run of smaller businesses too.
Prime Brokerage will be a shambles for years while creditors tussle over entitlements – with the UK end gutted of cash, there will be some splendid rows over who gets what; we know there will be a shortfall because in addition to the repatriation of funds to the US there was also a kind of custody run on Lehman by panicked hedge funds in the last few days. One wonder how good the books and records will be.
The real can of worms may be the global derivatives business (some of this overlaps with PB) as there will be a backlog of unconfirmed trades (and from the commenter above, trades with who-knows-what terms) going back years on really ratty cobbled-up systems. This is where the liquidator may have slipped up in not collaring the IT guys. Since his fee ranks ahead of all other creditor payments he may not be too bothered by a protracted liquidation, but he needs to be wary of creditors’ lawyers.
The fallout of the liquidation *may* expose how lighthearted management everywhere in big banks have been about their IT over the last 20+ years. But don’t bank on it: most of all, the liquidation is going to be a bonanza for lawyers, and they are in the business of delivering results for their clients, not critiques of bank management.
I was at Lehman in the legal department supporting derivatives a few years before the bankruptcy; some observations:
– There was a movement towards NOT documenting derivatives trades in the traditional “printed confirmation” format but rather, agreeing to one “form confirmation” and specifying that all trades thereafter would be subject to the terms of that contract. The trouble was, no one told the desks which were executing these trades. Therefore, a lot of trading systems probably reflect trades that were documented subject to entirely different terms.
– Most of the documentation unit was run by two non-lawyers who assigned other non-lawyers to review complex trading documentation, often supplied by Lehman counterparties; a lot of trade terms that would have been objectionable to a trained attorney or at least objected to if reviewed by the proper internal staff (Credit, Treasury) got through this way.
– The documentation unit were unwilling to challenge anything that the derivatives desks told them to do; hence even if a trade term conflicted with Lehman internal policies (i.e., contained a downgrade provision one level below the existing Lehman Holdings rating), the deal would get inevitably get signed with “business approval”.
– Toward 2006, the mortgage securitization desks started to really sit on the Legal department to have Lehman’s derivatives trading entities agree to whatever investors in their CMBS/RMBS wanted to see…if this meant no further payments to Lehman upon a Lehman downgrade below, say, A-, then so be it. Legal would sometimes get a pro forma sign-off from the relevant derivatives trading desk to formalize this waiver from policy, but generally, Legal was in the structured finance desks’ pocket.
– Anyone who tried to elevate these concerns internally with Legal, to the level of the GC or Tom Russo, Counsel to the Executive Committee was immediately shot down and told to get with the program.
– Most of the former Legal/documentation staff at Lehman are now with Barclays or the trustee for Lehman, Alvarez & Marsal. La plus ca change.
–
Derivatives contracts can be negotiated and formed by way of an ever-growing number of message channels — fax, email, chat, text message, social networking and more. The number and diversity of business messages that financial firms need to archive — just so they can know what their people did and did not agree to — is exploding. See discussion: http://legal-beagle.typepad.com/wrights_legal_beagle/2009/03/record-keeping-in-financial-markets-to-soar.html –Ben
SWEET! i love to see their greed bite them in the ass. all you tightwads working on wall st have it coming. either share the wealth or gtfo.
”
stuffed derivatives trades into existing fixed income trading systems
”
If you trace through each component within a fixed income account you will soon find that the underlying support of value often rests squarely on many flaky derivatives.
Would you guess that Lehman Executives had long since wanted to spin off the derivatives trading simultaneously holding on to some interest as silent partner? Did they hope for a spin off merely to disentangle the spaghetti wire connections that would eventually be so difficult to evaluate? Were the interactive connections changing each day that markets closed down?
Were the executives too far down the runway to abort the take off? Was it too late for a spin off? Were they caught up inside a Горбачёв dilemma?
Can this be prevented by splitting up the banks into one service corporations — one for deposit accounts, another for credit card, another for checking. Should the integration of functions be only in the hands of the customer but not in the hands of the professional banker, pseudo-fiduciary? Should deposit only banks hold only T-bonds and no real estate, no assets but T-bonds? In Liechtenstein you can open a deposit account based on any of the worlds currencies. Should American Banks give this choice to customer? Let customer decide, decide and control?
U B Judge
Yves,
Prior to changing careers to become a shrink, I was a software engineer and software business owner for quite a few years, and my company produced a few well known major application type packages. Having seen code by the millions of lines, I will say this: anybody who skimps out on documenting his code is a total moron doomed to failure. Being able to maintain code (debug, upgrade, customize, etc) is often a greater hassle and expense than writing the initial package, and that with adequate documentation. Without proper documentation it is utterly impossible to maintain a large system.
As far as the suggestion that they could have instead used off-the-shelf software solutions, that route may work for small mom ‘n’ pop shops, but with companies as complex as Lehman, it is impossible. You have to code it yourself in C++, Java, or something like that. You need an army of coders for a company like Lehman. MS Office just won’t cut it.
Signed,
Vinny G. – maverick shrink extraordinaire, legendary programmer emeritus, and unstoppable think-tank of one…
Under the guise of ‘we know its in there somewhere, if only we could find those IT guys..’, we remain on the sidelines of the major, unknowable question about these financial derivatives that really must be answered ASAP now.
What’s that stuff really worth?
As soon as we find those IT dudes.
Somewhere Prof. Donald Knuth‘s charitability is being strained …
(I’ll bet he’s read some Niebuhr.)
this seems gremain to the overall topic.
The unspoken truth about managing geeks
http://www.computerworld.com/s/article/9137708/Opinion_The_unspoken_truth_about_managing_geeks
also in the Lehman case there is a lot of the systems developed in the US used in europe and systems developed in europe deployed in the US… the new owners BARC and Nomura were only ever getting half of the DNA… not surprising they are misssing half the source code… they never had it !! thats what happens when you split a global bank in a weekend!!