By Bob Goodwin, an investor and medical device entrepreneur who lives in Mercer Island, Washington
It has been a good generation to be involved with software. The scarcity of the skillset combined with the demand for the output have generated outsized incomes, while the work has been consistently rewarding. Our quirky group of builders has had an outsized influence on our industries, not to mention our culture and ideals. But that influence is looking less benign as the rigid procedures of computing are changing commercial relationships and the application of the law.
In our prehistory, the 50s, 60s and into the 70s, there were at first a handful of computers and eventually hundreds of thousands in the world. In those days data was impossible to crunch, and yet data was impossibly valuable to crunch. Any means necessary included costly, limited and ill-behaved combinations of experimental equipment. Memory was literally sewn together by hand, with a magnetic bead for each bit, and a pair of wires running through it along each axis of a grid. Longer term data would be stored on a rotating drum, and the programmer would try to time the placement of the data on the drum so that data would come around about the time it was needed. I was a teenager in the early 70s, and took a non-credit system programming course at Dartmouth when I was junior high school. But my memory of the time was writing programs with toggle switches, and not keyboard, debugging by watching lights, and patching paper tape programs with tape.
The software engineer had the tedious job of primarily overcoming obstacles, while trying to work the magic of his trade of extracting value from large investments in equipment. The best in the field resembled master craftsmen who could build rocket ships from tin foil and masking tape, and make it look easy to realize the potential of the computer.
But the golden years only started when the computer became a tool for human efficiency. It was easy then because there was so much inefficiency in life, and a simple tool gave its user an obvious and immediate edge. This in turn triggered an avalanche of imitation which changed daily life. The change was not immediate due to limitations in cost and talent, but none-the-less it was a rapid and confusing transition.
An innovation at the time only needed the benefit to outweigh the costs. The costs were astronomical: expensive equipment, steep learning curve, frustration, and lost data. But the benefits, even with buggy software, were even greater. Because the greatest cost for the user was (and is) the learning curve, monopolies were built on being the first with an innovation, giving the second movers in the market an impossible barrier of retraining the customer base.
The software engineer, however, had no barriers. The credo of the day was be first, solve a single common problem for a single user. There would be plenty of time to correct mistakes later. We were called computer programmers then, and were young, smart, opinionated and fiercely independent. The cultural shock from the Vietnam War was still fresh, and young people lacked any identity that might have tempered conceit. There are elder statesmen of software today. Computers, like language and music, must be mastered young.
The vacuum of obvious applications was quickly filled, but under the covers of the trade were a hodgepodge of ideas, methodologies and standards, much like the early days of the railroad where every stretch of railroad line would have a different gauge and standards, and so goods would have to be stored and repacked at crossroads. The true inefficiency of those railroads was that neither equipment nor skills were fungible. The evolving railroad system had high levels of abandoned infrastructure, and a continual investment retraining and recruiting skilled workers. The same is even truer in the software industry. Very little thought was given to this problem, because value was always higher than cost, and demand for talent was always in excess of supply.
The next phase of software engineering involved the connection from A to B. Warehouse to customer, booking to airplane, student to university: the centralization of data. The computerization of the government functions accelerated in earnest at this point. This time still predated the internet as we know it, but computing was now about interconnecting, but not yet in the democratizing sense of the world wide web. This was probably the low point for our field, and a lot of practitioners from this day have been flushed away. All of the limitations of the earlier land rush brought disappointment when things always seemed broken or at least brittle. This was the era when a lot of larger initiatives in computing were canceled or scaled back, and there was a cynicism that industrial management techniques were squeezing the soul and creativity out of building software, and thus the bad outcomes were predictable.
But the problem was with the trade, not the business or management styles. Connecting A to B very soon required Connecting B to C, and so on. There needed to be an agreement on how things should connect generally. If A, B and C were thought of as business desks operating by phone in three countries in Europe, Then ‘B’ might need to be fluent in the cultures, traditions and currencies of three countries. It didn’t take very long where the alphabet extended beyond C, D, and E, and software engineers found themselves writing connection code using multiple protocols, and being ignorant of the subtleties of each interface. Perhaps in France nobody calls you if a problem arises, where in Italy, transactions are silently kept for a week and combined to make later processing efficient. Ignorance of subtleties creates defects that are not detected until they are propagated to other actors. Tracing root causes across a network is a skill yet to be mastered. It was not possible in many cases to trace back from C to B to A. You don’t speak French, even if they would let you rummage through their paper trail. In this hodgepodge, only came greater anarchy, and a grinding sense that while computing was inevitable, the creaking massive gears of technology were as likely to crush as emancipate.
It was the mid 90’s when a consensus emerged on how best practices needed for connectivity of computers. The big consensus with the world wide web, which was actually a set of individual conventions, such as TCP, HTTP, HTML, URL, IP, XML, and of course the browser and java-script. But there was much more, languages (Java, Python, Perl), coding standards, rich and elegant libraries of common functions. There is a massive list of technologies that are now universally accepted. There was a short window of time when consensus overpowered individualism, and then reverted. Of course the browser has blossomed since becoming the window to the cloud, but this would not been possible without those crazy letter acronyms strung together like pearls. This began the second great land rush, and another batch of billionaires were minted, and the connected world of computing was built on the monetization of cloud data.
Only at this point in the story can we see the seeds of today’s crisis. Once, in the not so distant past, the software engineer could invent something that replaced tedium, impacted nobody except the person buying the product, and could help the world without unintended consequences. When those opportunities were exhausted, innovation involved two agents, with only one paying.
A consensus emerged on a set of rules for how to create these new channels of control, but there was no consensus on the rules for the proper use of those channels of control. They were inevitably built to the benefit of the party paying. The value of new innovation was still so high, and the talent still so limited.
So society groaned but adapted as the computer began to assert rigid rules on commerce that had traditionally allowed informal resolutions between people. Commerce had not become more predictable, but the costs were being shifted by technology. We were getting richer fast because of computers, but these same computers were also used to change the relationship from the consumer to businesses, and often between businesses. Efficiency was accruing to the individual, but power was accruing the institutions.
Growing economies are very accommodating to uncomfortable change. Everyone alive today in America uses a computer in some fashion, and is familiar with its lingo, traditions and values. They also know that software is buggy, inconsistent, complex, frustrating and inflexible. If you go inside the beast it is obvious why. We had a big bang of consensus in the 90s. Those gains have already been monetized. Computing is unbelievably different than it was then. Computers are more than 10,000 times faster and bigger, and there probably 10,000 times more computers in the world. Each of those computers probably talks to 10,000 more computers than in the 90s, many trivial transactions on a home PC may aggregate data from 3 or 4 other computers and perform tasks that are entirely customized to you. For all the consensus we achieved in the 90s, the emerging realities have long since outstripped the implausible consensus, and there is a new vast bloom of inconsistency and confusion, far bigger than the last.
The growth of information, and the resulting mayhem, have left us unable to propagate the needed skills of our software craftsman. Even amongst our best craftsmen, it is only with great effort that we create a day-to-day consensus on how to keep the tracks connected and the trains running, without destabilizing the system. These are excellent craftsmen, but each is seeking reliability with different perspectives and different biases, and each has seen the consequence of error a great number of times. As has always been the case, the engineer is given wide berth, but the businesses must go on. Here to the solution has remained consistent through time. Throw more machines at inefficiency, and clump wads of software onto the side of existing clumps of “legacy code” that is too valuable to discard, but too scary to touch.
The brittle code works, mostly. And it works because it has to. When it breaks, everything stops. Software engineers who once might have worked whenever they were sober, started to wear pagers. 5 engineers sharing a pager would allow an engineer to live a traditional engineer’s life 80% of the time. But now there was a week that required sobriety. Sometimes this week did not allow sleep. Engineers are smart. They like to sleep. So the software works. Mostly.
What changed, and is subtle to even most engineers, is that we had begun to depend on software. It was no longer making us better, now we depended on it to always work, and the when software failed us, we failed.
Today we see the Healthcare.gov web site is performing far below expectations. This feels quite a bit like the 90s again. Spectacular gaps between expectations and reality are being exposed. Like the 90s most people are shocked, and root causes will only be understood after lengthy analysis. But in both cases there were plenty of insiders who were very aware of the vulnerabilities of these big projects, especially government projects, due to greater rigidity (for instance, in the contracting process, in legal requirements) and politicization. We are not expressive, because we don’t expect people to even know how to listen to software engineers. We can be pathologically optimistic or pathologically pessimistic, and we can be dismissive. But most of all we are just busy. It’s pretty rough out here right now.
Here I have to go into pure speculation, but I don’t feel far out on a limb. The issues are very consistent across the whole industry. We know that the goal of Healthcare.gov was to instantly present a taxpayer with a set of options provided by insurance companies, subject to state rules, and with subsidies given in real time based on government information residing in computers run by departments within the government.
I have seen sentences like the one I just wrote in power point presentations created by ambitious new employees. I know from the sentence, just like everyone else in the room that the useful idiot giving the presentation will not survive to usefulness. If it were a serious meeting, and there was a serious business requirement, we would politely untangle the sentence and expose the obvious. The existing infrastructure is almost certainly inadequate for the stated task. But even if the legacy systems could be miraculously used, the bloody lessons from building cloud infrastructure have heaped new layers of constraints of how to put together systems that have the kind of reliability, responsiveness, interconnectivity, latency and interactivity of the world class commercial web sites we use every day.
Again, I am actively ignorant. Given the same challenge, and a constraint not to rip up all legacy systems, I would have put together simulations and stress tests, and added redundant pathways and servers, and adding caching, and implemented predictive decision trees. A lot can be done with ingenuity. But they haven’t even built the payment system at all yet.
The fact that this is a government system makes it unique, and I know many are hoping that blame can be confined to government practices or subcontractors. And software engineering will be left on its pedestal. I disagree, with this emerging consensus, but let’s explore this.
The very elite tech companies have been a magnet for the best computer top talent for a long time. It is not this cohort that writes government software. Of course there are good engineers in every school, industry, country, and within each specialty. But there is no question that technical collaboration looks very different between engineers at a big tech company versus a contracting company writing government software. Government processes seem to always flow explicitly from written law or written policy. In other words, Government process are by definition implemented top down. But every well paid software engineer has had a lifetime of practice intimidating his bosses to let them work bottom up. All of the most brilliant engineers I have worked with would be emotionally contorted beyond recognition if they applied themselves directly to processes common in government software projects. Independent of process, government software contracts are not continuous, they are lumpy. So contractors need elastic labor forces. Elastic is not a magnet for skill. It should not be a surprise then that government software mostly works because the bugs were eventually removed, rather than because a master platform had been built to improve the efficiency of projecting law into code.
The Pareto principle, aka the 80/20 rule, gets an especially big vote when it comes to government software. There is always friction between features, quality, and cost/time. Features are almost always the casualty in government software because of the top-down implementation, but also because retroactive changes are tricky in a democracy. The fun part of the hidden Pareto clause, is that software engineers are really good at the analysis, and do it subconsciously.
We more to say about Pareto later, but the end result is code usually falls short of the law, The difference was made by a software engineer early via prioritization of work, but then only discovered after it was too late that features would not be available. Software engineers are not guilty of specifically overriding a policy based on their policy preference. But software engineers do have extremely strong preferences for how data and decisions are stored propagated and managed within a computer, and for the order in which work is staged. These preferences are mostly forged of pain, not ideology, but the result is that we are probably the least adaptable builders on earth.
One other issue is unique to government software. To the government, software is a capital acquisition like an airplane. Big tech has no similar illusion when they manage big projects. We like to say that nine women can’t make a baby in a month. The contracting company’s primary purpose in life is to implement a software project but sell it to the government as if it were a capital acquisition. The processes used in each type of acquisition are very different, but the contractor must create an entire cargo cult to modify appearances. Software eventually completes because engineers are actually professional and competent, but it is painful to watch the process, especially the executions afterwards in the public square when the boldfaced lies force you to look away.
For all the specific handicaps facing government software, the root cause of the problem is that complexity has overtaken convention. Healthcare.gov is just the dead canary. The ominous point is that while policy makers may have ceded control of their institutions to software, it is no longer obvious to me that even the software engineers have control of the software anymore.
It is 1995 all over again for the software industry. And it is bigger this time.
Something new is happening. Software engineers have happily disrupted the world for 30 years. But now we are being disrupted by big data. Computing has moved from PCs with floppy disks to data centers that cost billions of dollars and are mission critical to massive economic activity. The rate of data that is being stored and processed seems to be stretching the supply of talent thinner at an even faster rate. The amount of talent that is left for secondary or tertiary activity is shrinking. Code in 2000 implemented a plausible imitation of law. In 2020 we are too busy to fix it.
So now we know that software is far from a highly elastic and lightweight method of propagating policy and improving efficiency. Instead we know that it is concrete that hardens into strange shapes that quickly devolve into Detroit. But why is that? What is the root cause for these limitations inherent in software engineering? This question is not a bit new to us. This question has been studied and written about extensively. Tools and methods continue to evolve, but at a place slower than emerging challenges. I’ll address this issue in my next post.