The Glass Bead Game of the Root Cause
Last week, deep in the bowels of a failure report for a cascading DNS outage, I stumbled upon a line that has since refused to leave me. It was a note, appended by a senior engineer, a concession made after pages of technical triage: "Root cause: human error during a routine config push." We’ve all written it. We’ve all read it. It’s the period at the end of a long and frustrating sentence, a phrase that simultaneously explains everything and absolves us of the need to look further. But as I read it this time, I was reminded not of a server rack, but of a 13th-century cathedral.
The story, perhaps apocryphal, concerns the master builders of the great Gothic cathedrals. A particular builder, having meticulously erected the scaffolding for a new arch, gave an instruction to his apprentice. The young man, in turn, relayed the instruction to the labourer mixing the mortar. A miscommunication occurred. The mix was too wet. The arch, when the scaffolding was finally removed days later, showed a hairline fracture. It held, but it was a flaw, a weakness that would need monitoring for the rest of the cathedral’s life. Whose was the "human error"? The master for his imprecise language? The apprentice for his failure to clarify? The labourer for his poor technique?
The Gothic cathedral, like our distributed systems, was a feat of complex, interdependent engineering, built by teams of specialists who never fully grasped the entirety of the design. The master builder held the vision, the "architecture," in his mind, a kind of living blueprint. The stonemasons understood stress and load. The glaziers knew the behaviour of lead and coloured glass. There were no single points of failure, yet failure could creep in through the tiniest of gaps in process, communication, or assumption. Their "post-mortems" would have been long, chalk-dusty affairs held by candlelight, searching not for a single culprit, but for the weak link in a long and fragile chain.
The Seduction of the Single Blame
We, the keepers of modern systems, are too often like a frustrated master builder who simply blames the apprentice and moves on. "Human error" is a seductive endpoint because it is simple. It points to a discrete event, a faulty cog we can theoretically replace or retrain. But it ignores the system that allowed that error to become critical. Was the config change procedure opaque? Was there no dry-run environment? Was the rollback plan untested or, worse, non-existent? The error was the spark, but the tinder was laid by the system itself.
The true work of reliability, the boring, unglamorous work that defines our craft, is not in punishing the spark, but in continually wetting the tinder. It is in building systems that are not only technically resilient but also human-tolerant. It’s the insistence on peer review for every change, no matter how small. It’s the ritual of the pre-mortem, where the team brainstorms how a change could fail before it’s even executed. It’s the meticulous, almost obsessive, documentation that ensures the master builder’s intent is clear to every apprentice and labourer on the project.
Our cathedrals are made of logic and electricity, not stone and light. But the lesson from the old builders remains: a flaw is never just a flaw. It is the final, visible symptom of a hidden fracture in our processes. To simply label it "human error" is to walk away from the scaffolding, leaving the weakness in the arch for the next generation to discover. The real game is not to find who dropped the bead, but to understand why the pattern was so fragile that a single slip could unravel it.
Notes & further reading
A few pages I came back to while writing this: