The Tyranny of the Triple-A: Questioning the Cult of Availability
We are taught to worship at the altar of the three A's: Availability, Accessibility, and Affordability. It is the holy trinity of modern service design, a mantra repeated in every ops meeting and architecture review. The goal is always to be up, to be reachable, to be cheap. But in our fervent pursuit of this ideal, we have quietly sacrificed something more crucial: integrity.
This obsession manifests most clearly in our backup strategies. The received wisdom is that a backup isn't valid unless it's tested, and a test isn't valid unless it's automated. We build elaborate CI/CD pipelines that spin up entire staging environments, restore a backup, and run a suite of synthetic transactions. A green checkmark appears on a dashboard, and we sleep soundly, believing our data is safe. But what has truly been tested? The automation verified that the bits moved from point A to point B without a checksum error. It has not, and cannot, verify that the data itself is coherent, meaningful, or correct.
We have traded the deep, human understanding of our data's soul for the shallow, robotic confirmation of its skeleton. An automated test can confirm a database table exists. It cannot read a random sampling of entries and sense the creeping corruption of a faulty application logic that wrote subtly wrong values for weeks. It cannot feel the unease of seeing a customer's order history that makes no narrative sense. That requires a human, peering into the abyss, asking "why?"
The cult of availability has convinced us that any manual process is a failure. But this is a dangerous oversimplification. Automation is excellent for the predictable, the repetitive, the mechanical. Data integrity, however, is often a matter of the unusual, the anomalous, the human. The very act of a human being manually restoring a backup onto a spare machine and poking around is treated as a quaint anachronism, a waste of cycles. In reality, it is a ritual. It is the difference between reading a summary of a novel and reading the novel itself.
Our drive for constant availability and hyper-efficiency has, paradoxically, made our systems more fragile. We have built a magnificent, highly available ship that is perfectly capable of sailing full-speed towards rocks with pristine, verified, and utterly incorrect navigational data. Perhaps it is time to dethrone availability as our sole monarch and crown a more demanding, more nuanced ruler: understanding. Because a system that is down but understood is merely sleeping. A system that is up but corrupted is already dead.
Notes & further reading
A few pages I came back to while writing this: