Premature optimization is the root of all evil. –Donald Knuth
By Lambert Strether of Corrente.
Far be it from me to take issue with technical publishing guru and venture capitalist Tim O’Reilly; not only do I have a shelf or two of his great books, he puts cute animals on the covers! That said, I think O’Reilly’s “algorithmic regulation” is a horrible idea — absolutely Silicon Valley nutball triumphalism at its cringularity-inducing worst — and I want to explain why, as best I can with only an evening available. First, I’m going take a 30,000 foot view of “algorithmic regulation” and code and data; then I’m going drill down and look at specifics: St. Louis County, as it happens.
Algorithmic Regulation, Data, and Code
Here’s how O’Reilly (skipping the sharing economy uber-triumphalism) describes the relation between law and “algorithmic regulation” (apparently his coinage):
Regulation is the bugaboo of today’s politics.[1]
Laws should specify goals, rights, outcomes, authorities, and limits. If specified broadly, those laws can stand the test of time.
Regulations, which specify how to execute those laws in much more detail, should be regarded in much the same way that programmers regard their code and algorithms, that is, as a constantly updated toolset to achieve the outcomes specified in the laws….
Increasingly, our interactions with businesses, government, and the built environment are becoming digital, and thus amenable to creative forms of measurement [one hopes not “creative” in the way creative accounting is creative], and ultimately algorithmic regulation….
There are many good examples of data collection, measurement, analysis, and decision-making taking hold in government. In New York City, data mining was used to identify correlations between illegal apartment conversions and increased risk of fires, leading to a unique cooperation between building and fire inspectors.
It’s important to understand that these manual interventions are only an essential first step. Once you understand that you have actionable data being systematically collected, and that your interventions based on that data are effective, it’s time to begin automating those interventions.
[A] successful algorithmic regulation system has the following characteristics:
A deep understanding[2] of the desired outcome
Real-time measurement to determine if that outcome is being achieved
Algorithms (i.e. a set of rules) that make adjustments based on new data
Periodic, deeper analysis of whether the algorithms themselves are correct and performing as expected.
Open data plays a key role in both steps 2 and 4. Open data, either provided by the government itself, or required by government of the private sector, is a key enabler of the measurement revolution. Open data also helps us to understand whether we are achieving our desired objectives, and potentially allows for competition in better ways to achieve those objectives.
Before looking at O’Reilly’s four points, I’d suggest that “automating those interventions” translates into “code is law,” where law is the subject matter of the legal system, both statutes and rules and regulations. Naked Capitalism has been covering the metastasis of “code is law” for some time, and for those who came in late, I’d like to review the work:
Yves, inimitably, defined “code is law” as follows (June 2014):
[C]ode is law, the notion that underlying, well-established procedures and practices are revised to conform to the dictates of computer systems, with result being crapification of the activity.
In May 2012 (O’Reilly’s piece was published in October 2013[3]), I’d introduced the concept (“Code is Law: Literally”) after looking at Judge Elizabeth W. Magner’s decision in a MERS case, In Re Jones and said what Yves just said in longer words:
In summary, I’ve suggested there are two ways to look the foreclosure crisis:
- As a law enforcement problem, where banksters have committed illegal acts;
- and as a jurisprudence problem, where the oligarchs who own proprietary software systems have changed the nature of law itself: Code is law, not metaphorically, but literally.
In the latter case, what we understood the law to have been, in Civics 101, is then rewritten or crippled to conform to the code. Statutes, rules, regulations become vestigial. Code is the driver. (I can’t think another word for this than “revolutionary,” even if a revolution is “an overthrow or repudiation and the thorough replacement of an established government or political system by the people governed,” and not the people doing the governing.)
In September 2012 (“Code is law” Once More”) I’d suggested that this “revolution” was coming to pass. The Tjosaas family were foreclosed on not once but twice because of software problems. Their home and all their possessions were stolen, literally; Wells Fargo contractors drove up, trucked everything away, and “secured” the house. In each case, Wells Fargo’s IT system had given the contractors the wrong address. There were no criminal penalties or even charges. I concluded:
if we accept that “code is law,” then false positives from corporate computer systems are legal, just as they are in the copyright case above. Again, I’d argue that we don’t have a law enforcement issue, but a jurisprudential issue; the nature of law, and hence rule, has changed, as part of a larger change in the constitutional order that is still underway.
So now that we’ve seen actual examples of “algorithmic regulation” during the foreclosure crisis and the seizure of the Tjosaas family’s home, let’s go back to O’Reilly’s four point description of a “successful algorithmic regulation system” (to review). First, let’s take notice of a crucial omission. O’Reilly writes:
Open data plays a key role in both steps 2 and 4.
Remarkably[4], O’Reilly does not require “open source” code (that is, the software or program that works on the data); only data. False positives like the Tjosaas family experienced are inevitable in any system of “algorithmic regulation, ” since “testing can show the presence of bugs but never their absence.” But if “algorithmic regulation” code, unlike data, can be proprietary and secret — rather in the way that the chemical composition of tracking fluid can be kept secret — how are coders and their employers to be held accountable for bugs and induced to fix them? Will bugs even be tracked? Or is the occasional random loss of a home an acceptable price to pay for an algocratic utopia? (See The Economist on how open source makes medical software less buggy.) Second, remember that O’Reilly is advocating “automating those interventions,” and by “interventions” he most definitely means what we used to call “law enforcement,” because he writes about enforcing speed limits. So, you could be to all intents and purposes “arrested” by software, but if you believe — as the Tjosaas family believed, correctly — that the software is wrong — buggy, as all software is buggy — then how do you appeal? In the case of proprietary, closed source software, there is no appeal (at least in O’Reilly’s “successful” system) because you can’t see the software, so you can’t find the bugs.[5] Finally, O’Reilly’s work appears in an anthology called “Beyond Transparency,” and it’s worth underlining heavily how untransparent, how opaque, even obfuscatory, software programs are. Harold Abelson and Gerald Jay Sussman, with Julie Sussman, Structure and Interpretation of Computer Programs, p. 29:
Computational processes are abstract beings that inhabit computers. As they evolve, processes manipulate other abstract things called data. The evolution of a process is directed by a pattern of rules called a program. People create programs to direct processes. In effect, we conjure the spirits of the computer with our spells.
A computational process is indeed much like a sorcerer’s idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can answer questions. It can affect the world by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer’s spells. They are carefully composed from symbolic expressions in arcane and esoteric programming languages that prescribe the tasks we want our processes to perform.
Above, we focused on the code itself; the “symbolic expressions in arcane and esoteric programming languages.” In passing, we’d note that programming languages continually evolve, so O’Reilly’s “successful” system, whatever else it may be, is a permanent jobs program for programmers, either to maintain programs in old (arcane and esoteric) languages — like when all the grey-hairs were called in off the golf courses to fix ’70s and ’80s COBOL programs in the Y2K effort — or to rewrite them in new ones. No that there’s anything wrong with that, but let’s be clear that’s what’s going on[6].
More importantly, “algorithmic regulations” are no longer accessible to citizens. The rules and regulations of the law we understood the law Civics 101 might be complicated, but at least one could read the text, and lawyers and advocates could cite to it. But with “algorithmic regulation,” not only is the code that drives the automated intervention “arcane and esoteric” — if you think the Code of Federal Regulations is bad, try obfuscated Perl — and the very operation of the regulation is “much like a sorcerer’s idea of a spirit.”[7] How in good conscience can we ask a free people to obey regulations that they cannot read, when they cannot verify how the regulation was applied in their case, and when they have no way for appeal the “automatic intervention”?
From code in points 1 and 3, we now move to data in points 2 and 4 (and this will be simpler, I promise). First, “data scientists” are finally discovering that good data is hard to find; there’s a lot of labor in it. The Times:
“Data wrangling is a huge — and surprisingly so — part of the job,” said Monica Rogati, vice president for data science at Jawbone, whose sensor-filled wristband and software track activity, sleep and food consumption, and suggest dietary and health tips based on the numbers. “It’s something that is not appreciated by data civilians. At times, it feels like everything we do.”
Several start-ups are trying to break through these big data bottlenecks by developing software to automate the gathering, cleaning and organizing of disparate data, which is plentiful but messy. The modern Wild West of data needs to be tamed somewhat so it can be recognized and exploited by a computer program.
“It’s an absolute myth that you can send an algorithm over raw data and have insights pop up,” said Jeffrey Heer, a professor of computer science at the University of Washington and a co-founder of Trifacta, a start-up based in San Francisco.
In other words, we might look at O’Reilly’s piece more a sample of “Investor Storytime” for data wrangling startups in Silicon Valley, rather than as a serious public policy proposal, assuming the word “public” has meaning any more; open or not, the data doesn’t support what the techno-visionaries want to do with it. Further, given that law enforcement (“automated intervention”) is now data driven, malefactors of great wealth have every incentive to game the data they make available:
A new report charges that several oil and gas companies have been illegally using diesel fuel in their hydraulic fracturing operations, and then doctoring records to hide violations of the federal Safe Drinking Water Act.
The report, published this week by the Environmental Integrity Project, found that between 2010 and July 2014 at least 351 wells were fracked by 33 different companies using diesel fuels without a permit. The Integrity Project, an environmental organization based in Washington, D.C., said it used the industry-backed database, FracFocus, to identify violations and to determine the records had been retroactively amended by the companies to erase the evidence.
“Open” or not, how exactly does “algorithmic regulation” detect data that’s been doctored at source? Particularly when, as the Times points out, we’ve got great difficulty just cleaning the data to begin with?
So those are some of the problems with O’Reilly’s four points. But now let’s look at the real problem. It’s buried right in the first sentence:
[A] successful algorithmic regulation system…
But how does O’Reilly know what the boundaries of the system really are?
Premature Optimization and the St. Louis County
As everybody and their grandmother know, the engineers at Google have been developing a driverless car. BBC:
Google’s self-driving cars are programmed to exceed speed limits by up to 10mph (16km/h), according to the project’s lead software engineer.
Dmitri Dolgov told Reuters that when surrounding vehicles were breaking the speed limit, going more slowly could actually present a danger, and the Google car would accelerate to keep up.
Note that Dmitri is squarely in the mainstream of O’Reilly’s revolutionary constitutional order; it just isn’t a problem that he “breaks” the speed limit; his “automatic intervention” doesn’t override the law; it is the law. Dmitri’s problem is that he doesn’t understand the boundaries of his “system,” and so he’s prematurely optimizing it. We can see this very clearly by asking a simple question:
What about speed traps?
Let’s take St. Louis County — home of Ferguson — as an example:
Residents in the St. Louis suburbs didn’t believe the initial police account because many of them have had experiences with cops that were not happy ones. The area is riddled with speed-trap towns that collect half of their general revenue from traffic tickets. Instead of friendly lawmen, they see revenue agents with attitude and big guns. Police weaponry is increasingly powerful.
They see policemen as “revenue agents” because that’s exactly what they are. Here’s another example from the de-industrialized Midwest, in Cleveland:
More often than not, a tiny community—a village, an impoverished town, a designated special services district endowed with a certain degree of autonomy—will harness whatever police power it has and turn it into a source of revenue. I’ve explored this trend in East Cleveland, a impoverished inner-ring suburb of Ohio’s largest metro, which has struggled to raise revenue after decades of watching its middle class tax base dwindle to nothing.
Most law enforcement supports the cameras; so do private citizens who have lost family members to other people’s reckless driving. Meanwhile, the opponents of these cameras nearly always recalled the apparent history of moneymaking schemes; one Democratic representative evoked Elmwood Place as evidence of corruption, specifically referencing how 40% of the proceeds collected for tickets go directly to an out-of-state private company whose primary profit motive encourages it to issue as many tickets as possible.
So back to St. Louis. Why the speed traps? Well, they’re the outcome of a complicated history that has left St. Louis County very fragmented:
The map of St. Louis County, the home of Ferguson, looks like a shattered pot. It’s broken into 91 municipalities that range from small to tiny, along with clots of population in unincorporated areas. Dating as far back as the 19th century, communities set themselves up as municipalities to capture control of tax revenue from local businesses, to avoid paying taxes to support poorer neighbors, or to exclude blacks. …
The result of fragmentation today is a county whose small towns are highly stratified by both race and income. As blacks move into a town, whites move out. The tax base shrinks, and blacks feel cheated that the amenities they came for quickly disappear, says Clarence Lang, a University of Kansas historian who has studied St. Louis. Ferguson flipped from majority white to majority black so quickly that the complexion of the government and police force doesn’t match that of the population. That mismatch was a key factor in the tense race relations that contributed to the riots and, perhaps, the shooting itself.
That’s not all. Businesses choosing where to locate can play the tiny municipalities off against one another for tax incentives, prompting a race to the bottom that robs them all of desperately needed revenue.
There’s widespread recognition that fragmentation is holding back the economic development of greater St. Louis, but once a municipality is formed, however small, it’s exceedingly difficult to merge out of existence. Ferguson is comparatively populous at about 21,000 people. Many of St. Louis County’s postage-stamp municipalities have fewer than 1,000 people. Champ may be the smallness champ, with a 2010 population of 13, all white.
Now, the technical answer is to find the locations of the speed traps (they’re crowd-sourced), take the driverless car’s GPS location as input, and have the software slow it down when approaching known traps. Problem solved! But not so fast. Of course, the cops will game the speed trap locations, and speeds, but the real issue is that, like it or not, the fines from speed traps are important sources of revenue for poor and black communities like Ferguson.[8] Does Google really want to be responsible for further impoverishing (“cheating”) them — and for more shootings? Suddenly the clean boundaries of Dmitri’s system have expanded beyond the car, beyond the road, beyond the municipality, and to an entire urban area; in this case, St. Louis County, and a lot more people are involved than the drivers. Things have gotten messy. Things are no longer like Mountain View. The scope of the project is no longer clear. And since scope is the key to project success, that’s a big, big problem.
Problems with no clear boundaries are often called “wicked problems”. There’s a whole literature on this topic, but this description gives the flavor:
“Wicked” problems,for which there is no clear path to an optimal solution, no consensus on what an optimal solution would look like, and not even a clear definition of the boundary of the problems to be confronted (Rittel and Webber 1973), are unfortunately widespread in the management of social-ecological systems (Chapin et al. 2008, Jentoft and Chuenpagdee 2009, Peterson 2009)….In the face of wicked problems, when framing the problem itself is a political process, discourse structured by coalitions, ideology, and social practices can take on a central role in defining policy choices[9] …. As a result, power relations, emerging from formal institutional arrangements and informal network structures, can have an important influence on the way wicked problems of sustainability are framed and responses are defined (Chatterton and Style 2001).
In short form, we have politics to handle wicked problems; they cannot be handled algorithmically. This example is from fisheries (a common pool resource) but can be generalized:
Inspired by Rittel and Webber [Dilemmas in a general theory of planning. Policy Sciences 1973;4:155-69], it is argued that fisheries and coastal governance is confronted with problems that are inherently “wicked.” Problems are wicked (as opposed to “tame”) when they are difficult to define and delineate from other and bigger problems and when they are not solved once and for all but tend to reappear. Wicked problems have no technical solution, it is not clear when they are solved, and they have no right or wrong solution that can be determined scientifically. Instead, for wicked problems governance must rely on the collective judgment of stakeholders [for example, the people of Ferguson] involved in a process that is experiential, interactive and deliberative.
Here’s a list of wicked problems that cannot be solved via “algorithmic regulation,” including common pool resources and related systems:
In recent years, Ostrom and her colleagues have wrestled with how the IAD framework and the eight design principles can be “scaled-up” and applied perhaps in modified form to global common-pool resources and related “super wicked problems” such as climate change, the global food system, integrated management of water systems that cross national boundaries, marine fisheries, and other global commons dilemmas.
That’s a lot of problems! (And in this regard, it’s very telling that O’Reilly, when questioned on how “algorithmic regulation” would handle wicked problems, evaded answering.) And since I have succeeded in moving what O’Reilly and Google seem to regard as a tractable problem (driverless cars) into the wicked problems space, I would like very much to know how many other seemingly tractable problems meant to be solved by “algorithmic regulation” and “automated intervention” are in fact wicked. My guess is 80% and rising, although that’s not at all helpful from “Investor Storytime” standpoint.
Conclusion
Since it’s late, I’ll summarize briefly: I believe that “algorithmic regulation” is subject to bad and gamed data, accepts closed source code, and isn’t suitable for “automated intervention,” let alone the government of a free people. I also believe it presents problems as tractable that are in fact wicked, leading to project failure. Algorithmic regulation — from the standpoint of the social systems in which it is embedded — is prematurely optimized. It is, as I said, a horrible idea.
NOTES
[1] Not it’s not. Class warfare is the bugaboo haunting today’s politics. See this essay from Kareem Abdul-Jabbar. Regulation is a bugaboo haunting venture capitalists like O’Reilly. I know O’Reilly knows who Kareem is, but I sure hope kids these days do….
[2] Any time you hear “deep understanding” from a Silicon Valley venture capitalist like O’Reillly, check your bogometer; if it’s not pinned, recalibrate. Ditto for “innovative” and “disruptive.”
[3] Lambert blushes modestly.
[4] I say “remarkably” because many if not most of O’Reilly’s books support open source programs, and the languages typically used to write such programs.
[5] I suppose one could sue the software developer over an “automated intervention” in the form of a traffic violation, so theoretically there’s a judicial back door to O’Reilly’s “successful” system. But it’s hard enough to open that back door with human cops. How hard will it be with algorithms? I’d argue it’s so hard that it’s simpler to swallow hard and pay up, much as you would do with a cop in a corrupt, third-world country. Note, of course, that this dynamic actually incentives false positives, in cases where there are fines for violations of “algorithmic regulations”; “bug traps,” rather than “speed traps,” you might say.
[6] A permanent jobs program for programmers might be preferable to a permanent jobs program for lawyers. I’ve dealt with both; it’s a tough call.
[7] “But” — I hear you cry — “we will document the algorithm carefully!” But one category of bugs is a mismatch between code and documentation; it happens all the time, even in O’Reilly books. So which is controlling? The algorithm as implemented in software, or the documentation that describes the algorithm in human terms? If the documentation, aren’t we right back in the world of “manual intervention” that O’Reilly is so anxious to escape? And if not, haven’t we simply replaced “pointy headed bureaucrats” with “pointy haired bosses”?
[8] Here we are reminded of the $200 billion in title fees stolen from municipalities through MERS, when “automated intervention” nuked a land title system that had been “successful” for centuries.
[9] That would be why Google, unlike the people of Ferguson, maintains a presence on K Street, in Washington, DC.
UPDATE I just gave the post a light copy edit; “It was hard to write, so it should be hard to read.” For example, I meant to write “Silicon Valley nutball triumphalism,” not “Silicon Valley nutgall triumphalism.” Just goes to show that spell-checking is a flawed concept. Sorry.
Tim is a Genius at computer non-fiction publication. He is quite good at most technology thinking. Unfortunately, he falls into the same hole as any of the “Rationalists” and thinks in terms of computers. Computers and robots don’t sit on the toilet and think about ways to game the software they’re running. They don’t meet at some dark saloon to plot against the database management system or the operating system’s scheduler. They don’t turn out for meetings in public places brandishing blatantly illogical and inflammatory signs and yelling anti-software slogans. But people do all these things. And the big corporations, the politicians, and the lawyers of the rich and powerful do little else BUT game the system which makes Tim’s line of reasoning cute, but useless IMHO.
Robots can do all the things you describe if they are programmed correctly.
Which, of course, makes the situation even worse.
Apply adaptive malware (such as Russian mafia has been making for decades) to “algorithmic regulation” and things will go downhill really really fast.
Hey, Lambert.
Good piece, astute analysis.
Granted that a robot can be programmed to sit on a toilet. Then what?
Algorithms shouldn’t be in charge, people (under heavy surveillance) should. It can all be gamed. SEO is basically gaming an algorithm for your benefit. Why should anything else be any different? While SEO might affect your business it’s not going to decide to get you a ticket or choose between hitting a pedestrian and oncoming traffic. What if driverless cars can be gamed into massive traffic accidents by a pedestrian walking into a busy roadway and all the computers decide to hit other cars as that would cause less damage? Who is at fault? The pedestrian since they are the only “thinking” participant? Computers, like factory farming, are just a way to become a faceless entity in order to hide your corruption. “whoops, foreclosed/evicted people whose mortgage we don’t even own and aren’t even behind. Not our fault. Blame the computers.“
You are right that it is not the ‘fault’ of the computer. Rather, it is the ‘nature’ of the computer. By making the computer as the ‘law giver’, you are establishing a closed system with which you cannot reason any more than in the middle ages could the average person debate with the church and its’ law giver, “God”. The programmer becomes the new clergy. The nature of humans is messy. It allows for good and evil, but we are responsible for ‘us’. We should not turn over our responsibilities to those whose goal is power and wealth.
I think there’s nothing wrong with the government as software metaphor. The problem is in thinking govt can all be reinvented and current methods–which have been “debugged”over many decades or centuries–discarded. Indeed that was the whole premise of neoliberalism. Thus there was the notion that laws like Glass-Steagall could be put aside because we are so much smarter now. The truth is that most laws and regulations are on the books for a reason and were often hammered out as a result of bitter experience, aka “feedback.” One of the principles of good programming is code reuse. If it ain’t broke don’t fix it.
So the danger is not a “rationalist” approach but rather a false rationalist approach. In our scientific age science can deceptively be used to excuse many things.
Instead of ‘code is law,’ consider the current status of ‘law is code.’ From a technocratic standpoint, the maintenance of this code is atrocious.
ACA (Obamacare) is 2,000 pages of code that was known to be bug-ridden when it was passed. Since passage of a technical corrections bill is impossible, code errors are being fixed with ad hoc patches. In parallel, judicial review (Halbig v. Burwell) may pull the power plug entirely on the stuttering, error-spewing ACA system.
Meanwhile, other code such as the AUMF (Authorization for the Use of Military Force) remains in effect, with no maintenance and no sunset date. So does ancient, vacuum-tube era code such as the Espionage Act of 1917, used to harass the likes of Edward Snowden in 2014.
From a sysop POV, the federal code is a spaghetti monster of broken and often conflicting code. And that’s exactly how our rulers like it, since no one can predict how it will run.
Think of it as a Windows operating system…full of funky code snippets and patched together with chewing gum. I’m not saying that all of the moving parts of our govt have been debugged. Perhaps the reason the Espionage Act subroutine has hung around so long is that it was hardly ever used until Obie made the scene. The last time, I believe, was the Pentagon Papers case. So yes our code of laws could use some malware snooping.
Hear, hear!
Mayor: “Siri, our town’s revenue numbers are running below target for year to date.”
Siri: “I can help with that.”
Mayor: “What can you do?”
Siri: “One moment, please . . . there — I’ve set the town’s speed trap cameras to issue tickets at 34 mph instead of the posted 35 mph limit. Revenues will rise by 2.476% per annum.”
Mayor: “That solves it. But won’t people complain?”
Siri: “No one will ever know.”
First I want to say I think that it is correct to protest what has happened in Ferguson and because I don’t live there I don’t know ALL the rodent things the police and the justice system are doing …..
That said, I live in a upper middle class small town in the Midwest, and we have our problems with police too (not nearly as significant as Ferguson’s). Nonetheless, we have one major highway between our town and the middle sized city, Madison, Wisconsin. Most everyone drives it as we are a bedroom community and the people work in Madison. The speed limit in my neighborhood is the same as on the major highway. Second, our police are now stationed in neighborhoods to give tickets for cars who have one light out. Does this created goodwill…..I think not!
Yea at a certain point I don’t care if the driverless cars do deprive the poor minority or whatever cities of revenue, because it’s a really dumb revenue scheme. Funding government on fees instead of taxes is stupid, period. Oh and of course it hurts a poor person more to get some hundreds of dollars worth of a speeding ticket for going 10mph over than the speed limit, than a rich guy who gets the same ticket and for whom it’s pocket change.
An ex-gf’s father considered traffic fines just a cost of living well.
But Germany solved this problem. It’s only because we believe that winners are entitled to higher status that we don’t have traffic fines proportional to income.
This is an excellent, welcome, timely article. It is also the first occurrence in a long, long time that I see the concepts of wicked and tame problems being used in a discussion of current trends (I was introduced to the concepts some 25 years ago).
By the way, people enamored with algorithmic regulation and optimization would be well-advised to have a look at another mathematical regulation and optimization domain that was all the rage many decades ago: Operational Research.
Once, OR was viewed as the panacea to deal with every economic and social problem,
from scheduling shop floor work, through optimizing traffic networks, to city planning. Express the problem mathematically, and, no matter how many variables, branch-and-bound or linear programming would take care of it.
Practical failures accumulated, though; those of you working with ERP or MRP systems know that whatever optimal planning is produced by the wonders of OR does not consider many organizational factors and contingencies, and is thus to be taken with more than a grain of salt.
As a consequence, OR entered a deep existential crisis — historically marked by Russell Ackoff’s paper “The Future of Operational Research is Past” — from which it has not entirely recovered. Difficult to express everything in nice mathematical formulas when the reality is multifaceted, continuously evolving, difficult to quantify and with unclear boundaries.
I see the same fate befalling algorithmic regulation.
Operations Research is a great subject, but it does NOTHING to solve the modeling problem. If you don’t have an accurate model, you get an inaccurate result: Garbage In Garbage Out.
And you will never have a truly accurate model. Science is what we use to get models which are less wrong than the previous models. This means that you need to (a) do science, and (b) have OR applications which satisfy *robustness* criteria — as in, if the model is inaccurate by 10%, then the plan from the OR model is only inaccurate by 10%. We don’t know how to do that yet.
This is a terrible time for technology to be as sophisticated and advanced as it is. Because with the current human dilemma of near total corruption, internationally, in every nook and cranny of government, law, media and business, anything that can be used against us will be used against us (where “us” means the people with little say and “against us” means more rent extraction/control) by the kleptocrats.
Lambert’s arguments against regulation (of the herd) by recipe are more general and, I think, would stand even without the issue of systemic corruption, but that corruption, particularly it’s pervasiveness, is what makes technological schemes for handling large problems, such as automotive traffic, so untrustworthy and frankly, frightening.
And also frankly, there always seems to be a more simple solution not requiring super code-head algorithms that is never spoken of. For instance, It seems perversely appropriate that in all the public discussion about Google’s “driver-less” car as a means of solving congestion problems, no one in the main stream ever so much as whispers a word about the far more cost effective solution of more – and more effective – public transportation or smaller units of social autonomy (where work/home/recreation are more accessible).
I favor using software and robots to regulate our society, theoretically. Certainly what Tim O’reilly is saying is not silly and he makes good points–but, as implied by Lambert’s reply, we don’t live in a world where we can apply those principles without inviting even worse outcomes. The fact we are even talking about this, however, points to something larger, i.e., our political and legal process has broken down and is becoming, increasingly, arbitrary or simply favors certain narrow-interests who have captured the ear of regulators and lawmakers through some combination of bribes and threats. Regulations and laws are created through fundamentally corrupt processes whether you use algorithms or not the process is no good and requires a radical re-think.
At this point we ought to be very wary of “solutions” to our problems because we are in the process of deconstruction in our society. This process has to be undertaken more consciously and creatively so that we retain parts that work and throw out the rest. In the end algorithms and AI will be valuable tools but at the center of some new form of systems thinking there has to be communities. Part of building communities is to come together and set boundaries and rules and this we need to do at all levels.
Even in the best of times, software (and hardware), particularly where the source/mfg proecess is unavailable, reduces transparency and accountability when applied to the regulation of human affairs (such as counting votes) and should therefore sound alarm bells for any form of social control at any time. But, as you mention, now is a time where corruption is significant and this alone should disqualify even the best intended or the most logically argued software solutions – particularly those “imposed” on complex human issues since they will inevitably suffer the same corrupting influences that human agents are subject to, either in the way they are used, or more insidiously in the way they are developed and subsequently modified and tweaked.
One might similarly make a tight argument for the implementation of a nuclear power in terms of the current state of the art in security and safe operation. But these arguments are completely beside the point when that technology is under the control of a ruthless, corrupt segment of society bent always on maximizing profit/prestige/control over others regardless of consequence.
The above comment was intended to be in response to Banger’s comment.
I teach creative writing to many engineers and coders. Most of them want to write the next GoT. This is not true of all, but many write very detailed and precise fight scenes, with lots of swords flashing, and yet the scenes are completely flat, containing no emotion, only action. When I give them exercises to get beneath the skin of a character, to write from the inside of someone else’s body and emotions, they look at me and blink. They don’t know what I’m talking about. Again, this is not true for all, but many can’t think how to go about doing that simple exercise in empathy or, really, method acting; they can’t imagine doing it; they don’t understand why it’s necessary to do it; why the action fight scenes aren’t enough. Even when I point out the aching hearts in every GoT main character. It’s like I’m speaking Sanskrit.
Yet, for some reason, the ability to do this one thing — engineer circuits or write code — makes certain people (who are mostly white men) believe they are Philosopher Kings. That they cannot paint or sing or dance or make music or wrangle with literature or speak a foreign language or pilot jet aircraft or fix a compound fracture or ice skate or train guard dogs for the blind or negotiate peace between hostile spouses or just plain whistle — none of that matters. Because they can write code and therefore they, and only they, can instruct and re-order society, according to the logic of programming language and not human language and not the logic of painting or ice skating or whistling. Anything and anyone who can’t fit neatly into their binary world of 1s and 0s is simply non-existent. But the world isn’t digital. It’s analog.
So you are saying HBO’s Silicon Valley is more in the nature of a documentary.
O’Reilly’s sci-fi futurism also evokes other H’wood precedents: Robocop, The Day the Earth Stood Still.
Or Phillip K. Dick’s Ubik, where Joe Chip has to put a nickel into his refrigerator just to open the door. (And IIRC, the refrigerator nags him if he doesn’t go buy enough stuff, but my memory may have faded on that.)
Even if the source code was open and bug-free, and all data was miraculous anomaly-free, there are mathematical limits to the accuracy of pattern matching, regression, interpolation, and so on. There was once a general feeling that the advances of computer technology were going to be accompanied by advances in predicitive technology, but the most that happened was that techniques that were too complex to have been applied before became feasible. Doing things faster does not necessarily translate into doing things better, and to paraphrase Knuth, sometimes it’s better to get rid of data rather than try to amass as much as possible. So absent some breakthrough in the underlying mathematics, and not just some tweak to a centuries-old technique so it can handle some special case, the brave new world of algorithmic law and Rags the Dog ain’t gonna happen. And really, a computer has as much “spirit” as an adding machine, because that’s what it is, and anyone who’s used a decent debugger knows that you can indeed “see” inside a program and realize exactly what it does.
Corruption doesn’t mix any better with technology than alcohol with driving. And the people that control the software that controls us are very very drunk on both.
Very well-said!
This excellent post by Lambert raises some tremendously important issues. One basic problem is that, computer governed systems are hard to correct when they err, even with the best of will on the part of the humans involved.
Many (far too many!) years ago, I won the Lustig History Prize at my alma mater. There was a $500 award attached to the honor. Instead of receiving a check, I actually received a bill for that amount from the Bursar’s office with “Lustig History Prize” listed as a charge on my account! Even though everyone involved was very keen to rectify the mistake (unlike banksters who take advantage of computer “errors” to steal people’s homes) it wasn’t until I was in the midst of paying for my second semester books in graduate school that I finally received my money.
Computerization too often only adds another layer of Kafkaesque absurdity to people’s stressed out lives.
Yep, “computer governed systems are hard to correct when they err”. That and the fact that being hard to correct is oftentimes a feature in reality and a bug only in PR. Your experience with the prize money must have been frustrating but it was certainly apt to this discussion.
I think it’s important to distinguish, from a top level view, what we mean by “code is law” and also the algo cop/judge and let’s hope pre-approval thingy.
If you have a situation where some biased private entity creates a very much law oriented software system and it “surprisingly” interprets laws to the benefit of its creators, then it gets launched on the public and some time later found to be doing illegal things, then the upholders of the law shrug their shoulders and say “too late – baby here already” because complex software – no one can change software system – must bend legal system, then this is obviously a run-on problem.
Now there is great potential to do good here, and I would say even necessary. But Lambert’s Steps 2 and 4 have to take priority uber alles.
“some biased private entity creates a very much law oriented software system and it “surprisingly” interprets laws to the benefit of its creators”
We did this. It’s called MERS.
Ya, I know. I just abstracted it to a generalized class. Reusable that way.
That would be the Playbook!
Wicked problems? :)
The solution to any problem is eminently simple. If it can not be seen in this light, then the postulation is incorrect.
As people concentrate more and more on complexity, attempts to simplify create higher levels of utter bullshit, a lower grasp of reality, all in name of increasing mass productivity.
I see you’re not a gardener.
:) Share and share alike.
You are highlighting a serious problem. However, calling it algorithmic regulation sows confusion. The problem is not algorithms per se. There are multiple problems, I suppose, including proprietary algorithms, secret algorithms, and the fact that computerization can amplify errors. Also lack of sufficient human oversight.
We experienced algorithmic regulation a few years ago when we had a lot of work done on the house. The inspector came with a checklist (an algorithm) but we were prepared, because the contractor had used the same checklist when planning and doing the work. The algorithm was neither proprietary nor secret. There would have been no particular problem computerizing the algorithm, I believe. There is no particular problem computerizing the algorithms for tax returns, either.
“Yves, inimitably, defined “code is law” as follows (June 2014):
” [C]ode is law, the notion that underlying, well-established procedures and practices are revised to conform to the dictates of computer systems, with result being crapification of the activity.”
The problem is not the dictates of computer systems. The computer systems do not dictate anything. The problem lies with the people who have the computer systems designed and written with the aim of crapification.
Yes, computers are like the Good Soldier Schweik, who will do exactly what you tell them to do. That is why you need human oversight and override. The ultimate responsibility lies not with algorithms or computers, but with humans.
But “automated intervention” abolishes human oversight and over-ride very directly.
I wonder when it will dawn on the coders that these systems will be applied to them too? No doubt after they are built out for everything else.
For many software engineers, and software entrepreneurs, the lure of solving complex/cool problems overwhelms common sense almost every time. When it’s really cool stuff like automating an automobile in the real world, it’s impossible to resist turning every argument into a reason why the benefits will far outweigh the harm to society regardless of where the truth lies; but even for crappy stuff, like those horrific time (and now mood) management programs, it’s amazing how one can justify to themselves writing code they know will screw others. Except in the most blatant cases, they often actively ignore the fact that they will ultimately be affected like everyone else. When pushed, very very smart code-heads will make the infuriating but age old argument with a serious face that if they don’t do it, someone else will. Another particularly self serving one I’ve heard used in ernest is, “it’ll get us to the crash that much quicker”.
If I understand correctly, the idea is to automate interventions that humans are already making. (That was decidedly not what happened with MERS, the point of which was to avoid fees and skip steps.) But just as those human interventions are subject to oversight and override, so far there is no argument to omit oversight and override if those interventions are automated.
Yes, I realize that when the software is written by an interested party, the interventions change to favor them. So automation of regulations could make regulatory capture worse. OTOH, if the software is open to review by all stakeholders before it is implemented, and afterwards, it could make regulatory capture more difficult.
Another one from the silicon valley factory. They transformed us from a thinking natural being into consuming amoebas and now for the sake of a modest buck they will transform us to a golem. This O’Reilly guy doesn’t care about democracy, rights or reaching a consensus, he is an advocate for efficiency. This way of thinking brought us so far only disappointment and frustration. First and foremost ideas which put democracy in danger or directly or indirectly question it should be avoided.
Secondly, Godel’s first theorem of algorithmic consistency already gives a deadly blow to the idea, with or without human bugs. Coming from this industry I can assure you the the moguls of this industry do not hesitate to sell their products prematurely to gain “market share”, imagine what will be our fate in the hands of such software…dystopia…
Impressive essay from Kareem Abdul-Jabbar. Even more impressive that he managed to get it published in Time. I wonder if they read it first.
This begs the question.
How many lawmakers can read C++ or any other computer language – even QBASIC – for that matter? Here’s a free clue for members of the House of Reprehensibles:
include
As in, #include public.h ?
Thanks for “nutgall,” I had to look that up.
As I’ve mentioned here before, I’m a computer programmer for a state government crime information system. It must be behind the times because my co-workers and I spend a lot of time making sure the system complies with the law, and not the other way around. On occasion that includes staying late into the night or working weekends because a Friday afternoon court ruling says this or that data must (or must not) be displayed. More often it means working for months to make sure that everything is in compliance when some new or changed law is set to go into effect. My agency sometimes requests legislation when we think it important to collect/display information that the law does not currently allow, but we don’t process that information until the law says we can (which might be never). There may be places where code is law, but not where I work.
Lambert, brilliant piece in every way! A real classic!
https://en.wikipedia.org/wiki/Entropy_(information_theory) for those looking for some background on how this is inherently not possible (this goes back to 1948).
Anybody remember “Dune”? In it, computers were banned; the backstory involved an enormous war to get rid of them. Instead, they used humans to do some of the same things, resulting in some very weird people. But hey, it’s science fiction.
After reading Lambert’s piece, it’s suddenly NOT SF. He’s describing the prologue to that war, referred to in the book as a “Crusade.” We could also call it a revolution.
Technical note: Ferguson has almost exactly as many police as my town, which is twice as large and quite prosperous.
This tends to confirm that they’re regarded as revenue sources.
“Cringularity.” [pats self on back]