Yves here. Object oriented programming was supposed to alleviate some of the problems Goodwin describes. I wonder why it failed to live up to its potential.
By Bob Goodwin, an investor and medical device entrepreneur who lives in Mercer Island, Washington
Inside the software engineering field there are several huge paradigms with cult-like followers that continue to evolve in parallel with each other. Each seeks to master the process of developing software, as words like agile have been promoted in the English language from an adjective to a value system. But are any of these approaches adequate?
The goal of each of the processes is to make software production more predictable. Predictable has slightly different connotations in Computer Science, in that we are trying to extract useful information early and adapt to incomplete information elegantly. But from an outsiders point of view predictability means that the software had more of the correct behavior (features) and less of the incorrect behavior (bugs) and more consistency between how people who are collaborating measure good and bad behavior. Methods that are in vogue today seek fast adaptability and fast feedback on any divergence from expectations. Some of the practices that emerge might seem counterintuitive. Putting two programmers in front of the same screen decreases productivity by two, but causes such a massive increase in detecting divergence that it pays for itself. Writing all tests before writing the code that is tested is also massively inefficient, and its primary benefit is that is avoids the common case where a programmer believes the first thing he creates must be correct. Tests are less likely to hide bugs when they are written first, and when a test fails, the programmer is more likely to fix the bug than the test if the test is written first. Inefficiency is again trumped by improved feedback.
Predictability, to computer types, speaks nothing of schedule or on budget. Over 30 years we have gone full circle on how to talk about time and budget. As wild eyed hippies we confidently asserted our true belief that a completion time was unknowable, but was certain to get later each time you asked. Now as seasoned and practical professionals, backed up by scholarly data we say exactly the same thing. Our paymasters then were less technical, older and openly frustrated. Today they are the same, except younger and silent in their frustration. It is highly unusual to hear the word dollar come out of the mouth of a software engineer unless he was referring to a unit for a value of an item within a program.
In fairness, software is complex. Even simple software is complex. Most of the obstacles that are overcome are similar to the obstacles that a tradesman finds when he discovers that something won’t fit where it was supposed to and starts to improvise, or back off. But the software engineer is often working in a massive shanty town, and improvisations can take him up and down alleyways until he has a set of a dozen fixes that resolve a conflict without inducing excess risk.
It really is impossible to know how many obstacles will be overcome. Even in excruciating well thought out modules, there are unintentional and unstated assumptions. What if you had designed a building, and after construction began, were told it had to withstand the impact of a large airplane? Independencies that required reanalysis, would overwhelm you, before considering how disorientating it is to re-conceptualize something massive after it is half built. Even though software engineers feel like we get hit with that magnitude of spec change all too often, we usually pull it together given enough time. It’s just that nobody has learned how to simultaneously create software, and also know when it will be ready for customers. Big tech rarely manages to dates anymore, and when it does, it is usually with the explicit understanding that features will be sacrificed as necessary whenever the date is given primacy. Gone are the days when the boss asks for it all.
Again, the question comes down to root causes of the crisis. And this is where I am going to deviate from my peers and professors, and start speaking to people who buy or use the software rather than restating conventional opinions from our trade. What is wrong with software? What is the root cause, and what is the solution?
There are three problems with software. Please visualize a medium sized software project as being the approximate complexity of all the buildings, roads and Infrastructure of major city. The ‘typical’ unit in that city is probably a single family residence with a small lawn.
ISSUE #1. Need for building codes. Any building inspector or contractor could use standard tools to open up a wall in any house and will find 2x4s and sheetrock and copper pipes, electrical wire and 110V outlets. Different countries may have different conventions. There might even be some regional differences. But the convention of using 2x4s with a fixed spacing was settled at exactly the moment in history when housing became a commodity and the commodity became affordable and predictable.
Software engineering is currently organized like a massive shantytown in Bangalore, and I doubt there is a city planner in Bangalore who would even attempt to do a planned infrastructure update within a shantytown. Most everything in an American city is conventional across that city, and largely consistent across the country. There are lots of roads and apartments and shops. A few malls and factory, and the odd infrastructure items like sewer plants and flood control. The conventions get more specific as the infrastructure becomes more specialized, for example sewer plant, but convention still exists. Planners with expertise on single family homes are transportable across the country, and can make plans from their offices without ever consulting a craftsmen.
Software needs building codes, because we are building infrastructure that must be controlled by people who are entirely outside the field of software engineering. Software engineers can still build things that do not require building codes, just as people can build treehouses and experimental aircraft. But you probably can’t rent out a treehouse, and won’t find many takers for transportation via a blimp.
Building codes are needed to enhance the predictability of commerce. The very best module of software that is produced in the very best tech company today probably has building codes that its authors are very proud of, and they have certainty were a big part of the excellence of the package. But agreements amongst artisans on how to lathe consistent components is microscopically less interesting than conventions that unify all software. And the irony is that in the conventions of these excellent projects conflict with projects right down the hall, and a big chunk of the effort of our very best engineers is the conflict that arises from these small variances in convention. True reuse of code and having fungible talent are essential to fill the gap between the supply and demand for talent. This is only possible by creating building codes.
The benefit of standards also accrues back to the programmer, and this is where the biggest win occurs. Complexity quickly cascades, and consistency in patterns creates consistency in expectations and consistently in how pieces work together. There are fewer surprises to think about, fewer scenarios to analyze, fewer options to consider, and less opportunities for hidden problems to cascade to other pieces of software. There are a limited combination of ways to connect Lego blocks. While these codes are being developed, great effort is spent to analyze the consequences of each combination. If the choices and outcomes are enumerable, they can be remembered and consistent across the trade. This vastly simplifies process of examining options, and creating consistency of outcomes. There probably are 50 or 100 types of Lego blocks commonly used for single family homes, and maybe a different set for roads. A craftsman will still need to master his tools, but now has the confidence that following convention will reduce unexpected outcomes.
ISSUE #2. Standardize transparency. In software engineering there are three things happening in complete isolation from each other that are often dependent on each other. The fact that a software engineer has learned to work within these abstraction, can think about each independently, and make each work together is a testament to the skill. There is the code, or the software, which in our city is the buildings. There is the data that the software is using at any time, which in our city is the people and goods moving around. Then then there is the user interface, which has no parallel in our city, because the city is not serving customers through an interface. But perhaps if our city was Las Vegas, we could visualize our user interface as the Strip and its casinos.
To reduce the load on the engineer, and hopefully even allow non-engineers to regain control of their city, we need to stop making people work in the three abstractions one at a time. I cannot drive while looking at a map and also giving a presentation over a cell phone. I cannot do this even if all three are the same topic. Transparency is the ability to see everything in your city at any time. Transparency is also having all three abstractions working together in a way that gives non engineers visibility and control within their city.
Things go wrong all the time in every city. How does a city adapt to small problems? A problem with food can escalate to a health evacuation. A problem with transportation can disrupt a major show. The answer is that a city adapts because it watches for problems and escalates solutions. More precisely a cities people do all of this. A person can look in a kitchen and see what is missing, go down to a store and see what they have, and fill the gap. Nowhere in this corrective action was there a requirement for a craftsman to modify a building to fix a problem. Transparency allows a person responsible for a process to buy milk, and give it to the kitchen. People are the real problem solvers, and their absence in large software systems are a large reason why they are so brittle and prickly.
Organizing software for transparency is done explicitly to allow normal people to view and control the movement of data through consistent patterns of buildings and roads across an entire city. It is especially important that this visibility and control allows regular people to directly impact the user experience.
Issue #3. Isolation of complexity. There are always specialized and intricate pieces in any city and in any project. An auditorium and stage may some complex special effect. A tall building may require special elevators. The sewer system may have unique technology. Any person in a city would naturally be able to distinguish between the conventional and the special. That same person would also assume that it was special because it could not be conventional, but was important anyway. Lastly that same person would automatically defer to an expert on all things special, while happily make assumptions about all things conventional. The same must occur in software engineering.
Complexity should be explicitly made impossible when using Lego blocks according to building codes. Anything that is not built with Lego blocks should be considered a very dangerous black box, which could not easily breached accidentally.
All three recommendations I have given above are certain to happen in my lifetime, but I am pessimistic they will be driven from within. External policy makers are slow by design, wanting to build consensus on both the problem and solution, and to ensure that policy can be implemented. At least now we have a name for the problem. “Code is Law.”
In the next section I am going to discuss government policy and how it is impacted by the challenges facing software engineering.