Errors Inhibiting Complex Software

“The Atlantic” recently published an article entitled: “The Coming Software Apocalypse” by James Somers. It argues convincingly that humans are failing in our ability to create software systems that deal adequately with the most complex systems (nuclear arsenal, aircraft, radiation therapy dosage machines, etc.). “We used to be able to think through all the things it could do, all the states it could get into,” noted Nancy Leveson. We are now quite beyond that recalling a famous programmer saw: “When I wrote this only God and I could understand what it did; now only God knows.”

In this post I would like to try to define a collection of vulnerabilities in thinking which contribute to our difficulty creating software that’s sufficiently robust to be trusted in complex situations. Many of the vulnerabilities are built upon one another and / or are exacerbated by each other.

In this post use of “article” should be taken to refer to “The Atlantic’s” piece.

Vulnerabilities:

• “Bug”
• “Robustness”
• “Monoculture”
• “Priesthood”
• “Indeterminacy of Correctness”
• “Documentation and Interpretation”
• “Failure to Archive”
• “Fatigue / Sheer Scale”
• “Number Blindness”
• “Bad News”
• “Lack of Imagination”
• “Security”
• “Extrasystemic Environmental Change”

TL;DR Given these, I’m highly skeptical of humanity’s ability to avoid a software catastrophe as so many of these limitations seem tied to the natural hardware limits of our savannah-striding hunting and gathering optimized hardware.

Given these flaws we are unlikely to deal well with climate change, overpopulation, water scarcity, nuclear weapons management, biological weapons management or malevolent strong AI. I’m pretty bearish on our species. As such, the only reasonable stance is, in my opinion, Camusian existentialism.

Vulnerability Class: “Bug”

In the article’s opening horror story, phone calls to 911 in the entirety of Washington State were given the busy signal. The error: programmers coded in a simple space constraint to the number of 911 calls that could be placed. When this number was reached, no new calls could be allocated and thus 911 was “down.” While such a limit might sound inherently stupid to a layperson (“Why would you code such a thing?”), there are always limits that software has to negotiate in order to work within hardware and even cost parameters and it’s up to humans: Elevator-farting, “why did I come in this room again,” name-of- daughters’-friend’s-mom-forgetting, humans to make those calls.

Sometimes we simply get it wrong. We don’t hit the brakes early enough, we think we can make it home before the rain starts, we forget dental appointments. We just make mistakes. It’s the most banal, but also the most common.

Vulnerability Class “Robustness”

As I wrote this piece, this happened. Photo by Daniel McNeely

On top of the software design error, there’s a systems robustness failure. As profit-grubbers and / or “taxed-enough-already” electorates, we have chosen, and pressure our elected officials to trade-away robustness for short-term cost savings. Whereas a philosopher-king or an enlightened-dictator might have opted for multiple layers of infrastructure robustness to hell with the cost (e.g. multiple city walls), our democratic / capitalist system doesn’t honor fiat save few exceptions e.g. wartime allocations for total war, Manhattan Project.

To the article’s example, historically we had defense in breadth: small municipalities used local 911 services, run by locals and local infrastructure (e.g. phone systems). These systems were recognizable by their emissaries and practitioners. These agents of the systems were seen in grocery stores or at the PTA, their phone numbers were had by friends, their houses known to people in the area who, in turn were known to others. In the event of a local 911 failure the right people could be found, the local ’larum struck, and remedy found.

Under this folk model, a neighboring police district could be called and they could use external contact means to reach back into the blocked 911 zone or could order the phone company to direct 911 calls to them as a stand in.

Additionally, Citizens Band (CB) radio and ham operators used to form a reliable militia-level mobilization communications backbone in event of disaster. I recall my former Cisco Colleague Ian Kluft doing radio operator drills in and about San Jose. In the days after 9/11 it seemed downright heroic and prescient.

But these folksy-sounding designs have been traded to Mammon. In the Washington case, the processing was and is now done at scale, remotely, in Englewood, Colorado: 3 states away. There is no localized next-best service which while not as good is at least pretty good. We go from 100% service to 0% service. In the moment of 0% service, the roundabout efficacy of “calling your son’s elementary school teacher who goes to church with the sheriff and has his phone number” seems like paradise.

Vulnerability Class: “Monoculture”

Making complex engineering possible is also stymied by emergent monocultures. We continually hire tribal or non-diverse identities that have or converge to near homogeneity in thought. Many devalue diversity (or outright deny its value). A recent PR snarl for Google involved an engineer publicly broadcasting such a view.

While I can argue the rightness of the moral case of diversity-based hiring, from a cold-hearted pragmatic perspective, diversity helps groups see around groupthink and blind spots. Yet as a technical industry I consistently have seen and continue to see caste affiliation undermine creativity and thus heterogeneity of thought. Even where the hires are race / gender / sexual-orientation “diverse” their thought systems and / or pedigrees are as standardized as the radius of an Oreo.

ASIDE: This is no way supports that “all ideologies are equal” and that “everyone should get a chance.” If X’s white supremacist activities and advocacy enter my workplace, the chilling effect to uncountable employees and cost in litigation simply won’t be worth it.

Consider:

• “Xooglers” (ex-Googlers) like to hire Xooglers because they have a similar default training for dealing with hard problems. “I’m not hiring them because they’re like me, but because they’ve gotten a similar set of skills”
• Google shows preference for Stanford CS graduates. I’m sure they also thought that those hired weren’t being hired “because they’re like me, but because they’ve gotten a similar set of skills”
• Stanford, for all its pursuits of diversity in education, enforces its own conformity expectations.
• Venture Capital, cozily up the street from Stanford, is itself helmed by many Stanford grads and has created a host of success stories from Stanford alumni. This creates a pattern on which to match “will be successful” as “Stanford graduate” or “Xoogler.”

The sum emergent creation from these factors is a pipeline that defines a group, teaches it how to solve problems in a certain way (at school, in job tasks) and rewards those who have excelled in these processes by giving them the chance to create their own endeavors that will, in turn, repeat those inherited, common processes.

Let me add, this is not to pick on Google, Venture Capital, or Stanford per se. I, and the world in general, benefit from their work every day. The names and schools could easily be swapped out based on different countries, companies, universities, historical eras, etc.

ASIDE: Funny, my (Google) Chromecast is cycling pictures in a screensaver. It just showed a beautiful Mission style colonnade: an unmistakable walkway through…Stanford.

While it is almost a throwaway line in “The Andromeda Strain” film of 1971 the homosexual, childless character notes that these intersectional identities are to the advantage of the project: she won’t go along with patriarchy-backed bully males out of respect for gender roles or sexual desirability. Another researcher is chosen for his unmarried, childlessness. Crichton understood the game theory might have guidance for building better teams out of diverse backgrounds.

Vulnerability Class: “Priesthood”

This is actually a side-effect to monoculture, but I think it’s different enough to merit its own class. Given a monoculture that finds itself controlling vast monetary resources and political capital, the monoculture will begin to myth-build itself. This is the phase when “God-Chosen,” or “blessed” or “rightful” tends to define the monoculture’s conception of itself. The tribalism moves from “one of many tribes using these resources” to “our tribe being the best steward of these resources.” As this progresses, the monoculture moves to priesthood.

Given the economic and social benefit to being in the priesthood (peasants bowing and scraping, tax-free status, gold utensils, profit from sale of indulgences, an adorable post-war 2 bedroom in Palo Alto, etc.) it becomes beneficial to increase the barriers to entry of the priesthood. To justify the stymieing of the strivers and the rightful elevation of the priesthood it works to block would-be entrants to the priesthood’s former levels.

In its benevolent form, priests share knowledge of bee-keeping, bridge-building, and beer-brewing with their regional flock (e.g. Belgian abbeys teaching agronomy). In its malevolent form the priests use and abuse the townsfolk for personal enrichment. The least moral of the priests see “their knowledge” as being something “reserved” for them and that the plebs should be proscribed from pursuit of that knowledge. They then introduce (either intentionally or unintentionally) barriers to common understanding. The model I have is forcing themselves and institutions in as a middlemen between the raw artifacts of the religion and the parishioners.

The Catholic church resisted the laity [interpreting scripture on their own][laity] from the primary source. Perhaps this was owed to concern about schism or the welfare of the parishioners’ souls. The reason they feared it so, I suspect, is that without proper inculcation and seduction by the power and privilege that comes with membership in the Priesthood (being bought off), the lay population might throw off the shackles of their bondage and upset the revenue stream and influence the priesthood had come to enjoy.

Few rational agents can look at the multitude of contradictions present in the Bible and conclude that it was of one piece, Holy and perfectly made. Even on simple facts of characterization it’s so contradictory as to warrant suspicion e.g. “Was Moses a Good Speaker”. Consequently it’s in the Priesthood’s best interest to hide, squelch, and misdirect flaw-revealing behaviors undertaken by “outsiders.”

The laity questioning the theological contradictions and social contradictions of the Bible would show its internal rot and, having done so, would present avenues of attack against the Priesthood’s lives of privilege. To the technological parallel, the shibboleths of the “Monoculture” ensure the Priesthood’s elevated status. When a priest, like Luther in his time, questions the rectitude of the Priesthood, they are often shouted down or excommunicated.

Vulnerability Class: “Indeterminacy of Correctness”

From Liquid Haskell

The National Highway Traffic Safety Administration enlisted software experts from NASA to perform an intensive review of Toyota’s code. After nearly 10 months, the NASA team hadn’t found evidence that software was the cause — but said they couldn’t prove it wasn’t. … team demonstrated that there were actually more than 10 million ways for the onboard computer to cause unintended acceleration. They showed that as little as a single bit flip — a one in the computer’s memory becoming a zero or vice versa — could make a car run out of control.

Recognizing the common idiom “it doesn’t take a rocket scientist to do X,” here the code that controls a car was impossible to fully understand by a team made up of rocket scientists!

For many software programming languages proving that the code is “correct” is impossible. This may account for some of the rise of functional languages again that arose out of set theory; those languages offer the means for evaluating correctness of code. The code could be proven, like a symbolic logic proof, to be correct.

Although, this doesn’t meant that the code is bug-free: the requirements could be wrong. Bugs such as:

If the gloves fits, give the King of France a cigarette.
The glove fits
Perform action
Does the King of France have a cigarette?
Expect: True
Result: System error: There is no King of France at present.

still arise in (formally; morphologically) “correct” programs.

Vulnerability Class: “Documentation and Interpretation”

This is almost a banality. Human communication is hard and imprecise and it’s very hard to communicate with precision. Divorce, self-help, business communication, etc. media market size stands as a testament to our absolute hopelessness in this regard. I quote from the article wholesale:

Typically the main problem with software coding and I’m a coder myself, Bantégnie says, is not the skills of the coders. The people know how to code. The problem is what to code. Because most of the requirements are kind of natural language, ambiguous, and a requirement is never extremely precise, it’s often understood differently by the guy who’s supposed to code. … On this view, software becomes unruly because the media for describing what software should do—conversations, prose descriptions, drawings on a sheet of paper—are too different from the media describing what software does do, namely, code itself. Too much is lost going from one to the other. The idea behind model-based design is to close the gap. The very same model is used both by system designers to express what they want and by the computer to automatically generate code.

Vulnerability Class: “Failure to Archive”

When software is created the code goes into systems for tracking the code’s evolution and changes. I do not believe that the paper documents, minutes from the meeting, photographs of the whiteboard are equally well-preserved. They are considered to be acceptably ephemeral because the only artifact that “counts” is the code.

But this idea would be absolute nonsense to a chemical laboratory e.g. Dow or Big Pharma. The notebooks written in are property of the company and are archived. Software, in my experience, rarely shows such assiduous record keeping. As such if the code repository were to be destroyed (lack of “Robustness”) the knowledge to re-create it is gone and the bugs once hunted and squashed would be re-introduced and new bugs created alongside!

In a few sectors this is not the case e.g. the FAA where all changes are required to be traceable to changes in requirements documents.

JK Rowling beautifully suggested this in the building Hogwarts which was created by powerful and ancient magic that no one understood anymore. In addition to providing a fascinating tool for plot leaps (“Who the hell put a Chamber of Secrets in here!?”), it describes what happens when “magic” (or as Arthur C. Clark called it “any sufficiently advanced technology”) doesn’t have its history catalogued. In many ways this also recalls the crimes of the destruction of historical monuments of one priesthood by another e.g. Catholic priests scraping ancient vellum for writing material and losing Archimedes' near-invention of Calculus; Catholic missionaries burning Mesoamerican documents; Taliban destruction of Buddha statues, et al.).

Vulnerability Class: “Fatigue / Sheer Scale”

According to MIT research, active working memory in the human animal is 4.

Thankfully we can “chunk” 4’s into 4’s and those into 4’s of 4’s. We can remember eggs, milk, diet coke, and Mike & Ike’s but toss in a dill pickle jar, hoisin sauce, or Meg’s lunch order from “When Harry Met Sally” and you’re very likely to forget something.

Now apply this to writing code. Part of what the audience gets about seeing Charlie Brown lug around “War and Peace” is that it’s a soul-crushingly big book. It’s a book that a bald-headed boy nearly has to grab a wagon to pull. “War and Peace” weighs in at over a half-million words. For reference, “Huckleberry Finn” is about a fifth the length.

Imagine you wrote one word from “War and Peace” on a notecard and made them insto a stack. The stack’s height (or length to prevent it from toppling over) is calculated as: ( 7*10^-3 * 587,287 / 12), 34 feet, over 5 LeBron James. While the task is daunting each card’s work is easy: read a word and understand its meaning in the context of a sentence.

Wait, did you catch that? I made a math error. If you didn’t catch it, ask yourself why? Big numbers with funny notation might have distracted you. The fact of the matter is that given note cards of 0.007, each with a word from “War and Peace” on it would stack to be over the length of a football field! Now you have some appreciation of how hard numbers at scale are to work with and to keep mental track of. Keep that in mind as we continue.

Now imagine that one line of code was on each one of these card instead. Now instead of “the” or “revolution” it’s: if (speedState[n-1] > weatherAdjust(currSpeed) + epsilon) {. You might need to go back 50 cards to remember the definition of weatherAdust or need to scribble down the state of speed in the previous lookup, etc. The work on each card is now hard. Not knowing anything else about the code save its language and “note card stack height” is spirit-wilting. Just thinking about it made me consider getting a beer.

But here’s the thing: car code bases are multiple factors larger than “War and Peace” by, in some places, twenty-fold. So now imagine 20 football fields stacked with lines of code. How excited are you to leap into that code and trace the logic? How authoritatively could you say you understand all the unique states. Do you remember the second item on my grocery list above? Probably not, you’ve already cleared that cognitive scratch space. That resource versus 20 football fields of code? I’m not betting on my species-mate.

A related problem surrounds calculation of transitive dependencies. If A requires B and B requires C; C is considered a transitive dependency. In software it’s simple to think of oneself as A and to know that if B changes your code in A might be impacted. But it’s near impossible to calculate the exposure in A-land of changes in C or C’s dependencies or D’s dependencies, and so on. A recent meltdown in the NodeJS ecosystem happened because someone removed a widely used dependency for padding strings.

That’s right: whole sites through the snarl of transitive dependencies were rendered DOA because of a removal of a library for padding strings for alignment.

Vulnerability Class: “Number Blindness”

This is an error which we saw at play in the previous section when I made my math error: when faced with large numbers we have trouble quantifying and imagining what that number means. What does a million dollars look like?

How much money is present in this picture from “Breaking Bad?” We have very little capacity for really grasping whether this is a million or not. Descartes pointed out this difficulty when he wrote about the difference between imagining and “intellectually understanding” a Chiliagon (a 1,000 sided polygon). Who would call that polygon a “circle?” Or how would one know a chiliagon versus a myriagon (a 10,000 sided polygon). We’re fine with a triangle, rectangle, hexagon, octagon, but what about a pentacontagon (50-sided). We’re hopeless. Because of our inability to understand numerical shifts beyond the 1-20 or so range we can’t appreciate exponential effects (climate change) or appreciate when we’re off by infinitesimal or gigantic factors. We’re clueless. Could you spot-check air traffic if a chiliagon was an essential component to the optimal route? Heck no.

Consider climate change to be another example.

Vulnerability Class: “Bad News”

“The rock is gonna fall on us,” he stood and told the class
The professor put his chalk down and peered out through his glasses
But he went on and said; “I’ve seen it, high up on the hill
If it doesn’t fall this year then very soon it will!”

Crazy boy–
Everybody knows the rock leans over the town
Everybody knows that it won’t tumble to the ground
We’ve more important studies than your fantasies and fears
You know that rock’s been perched up there for a hundred thousand years

…
Just a madman–
Everybody knows the rock leans over the town
Everybody knows that it won’t tumble to the ground
Everybody knows of those who say the end is near
Everybody knows that life goes on as usual round here

First, I mourn the death of Harry Chapin. I weep for the death of his tenderness and hope and humanity. He was too good for this world. Secondly, I love his New England brahmin accent.

Second, Chapin perfectly captures that anger, exacerbated by number-blindness in which the likely bad outcome is ignored for the more pleasant outcome by those (Priesthood-members?) who stand to benefit from the status quo.

Surely the volcano north of Pompeii won’t bury the town. Surely I don’t have an incurable disease. Surely no one is going to use a plane as a missile….

Piteously wrong.

Surely life isn’t this miserable and there must be an afterworld. Surely my ability to have affordable, master-planned, air-conditioned comfort at the end of highways that take my fossil-fuel burning buggy to Sam’s Club where I can buy cheap Chinese serf-labor produced T-shirts and back isn’t contributing to cooking us and every other living thing including all my loved ones, giraffes, and my dog alive. Surely some force is out there looking out for us. Surely there’s not threat from beyond the stars.

Unknowable, but the evidence leads to bad news. And we don’t like bad news so we either, as speakers, underplay it, understate it, or, as listeners, filter it. And yet sometimes it’s the critical datum we need but that we refuse to integrate.

Vulnerability Class: “Lack of Imagination”

convinced that car manufacturers weren’t taking software flaws seriously enough, demonstrated that a 2014 Jeep Cherokee could be remotely controlled by hackers. They took advantage of the fact that the car’s entertainment system, which has a cellular connection (so that, for instance, you can start your car with your iPhone), was connected to more central systems, like the one that controls the windshield wipers, steering, acceleration, and brakes (so that, for instance, you can see guidelines on the rearview screen that respond as you turn the wheel).

Here software companies who put code in cars (formerly known as “auto makers”) clearly failed to imagine an attack vector to the core code (steering, acceleration, etc.) via the entertainment system. Stronger sandboxing of code, data validation, etc. was required.

This was a failure to consider “Security” and lack of robust “Requirements” doubtless abetted by “Sheer Scale.”

Vulnerability Class: “Security”

I’m not sure this is an atomic vulnerability, I think it may only be emergent as a compound from others.

This is a catch-all which integrates most of the other failures as a mega-failure. Software developers design to get the code formally correct and requirements correct (“Interpretation”). Most requirements authors write things like “be secure.” What kind of security (“Robustness”)? Confounded with “Lack of Imagination” errors which are exceedingly rife in “Monoculture” this class is to be expected to emerge.

Vulnerability Class: “Extrasystemic Environmental Change”

This error comes about when programmers can only consider the environment in which their software exists today. But rapid change in the environment is possible. Ice sheets are melting and assumptions about “hard truths” are no longer true in staggeringly short order.

In this 911 system’s case, the programmers had set the number to be in the millions. How many millions of calls will be clocked? It’s hard to estimate. Suppose your sales team tells you that you’re signing up communities of 1,000 people. Millions of calls won’t be reached before the “sweep” process runs; life is fine. So as a programmer you cap the MAX_CALL_COUNT number at 5 million so that the municipality can afford the service contract and marginal hardware costs.

But now suppose massive success happens. As an engineer do you get a sales notification every sale? Is the sales team reporting to you? Little do you know your little company is starting to sign municipalities and has landed a few big fish. Suddenly your limit is capping communities for which it’s not appropriate. Your “hard coded” value is now a ticking time bomb because changes outside of “build the software” don’t roll back into the software’s maintenance.

Conclusion

Given these flaws we are unlikely to deal well with climate change, overpopulation, water scarcity, nuclear weapons management, biological weapons management or malevolent strong AI.

• software
• leadership
• programming

• Next: Mobile Upload: VC Conference OR Kid Pick-Up at Palo Alto High?
• Previous: The Buried Complexity of Software