The Necessary Glitch: On the Unreliability of Our Own Backups

We treat our backup systems like sacred texts. We whisper incantations over them, perform regular rituals of verification, and stake our professional reputations on their infallibility. The common advice is a mantra of absolute trust: automate, encrypt, test, repeat. Build the fortress so strong that disaster becomes a mere inconvenience. But I want to propose a heretical thought: this pursuit of perfect reliability in our backups can breed its own, more insidious form of failure. The greatest risk is not that our backups will fail, but that our conviction in their perfection will render us incapable of handling the one failure we didn’t plan for.

This isn't about a technical flaw in the backup script or a corrupted archive. It’s a human flaw, a blindness born of comfort. When the system hums along for years, when every restore test returns a green checkmark, our vigilance begins to atrophy. We start to believe the map is the territory. The pristine, automated snapshot becomes the de facto state of our systems, while the messy, evolving reality of production is seen as a temporary deviation. We build a beautiful, idealized fossil record and forget that the organism is still alive and changing in ways our fossilization process cannot capture.

The Illusion of the Known Good

The concept of a "Last Known Good State" is the cornerstone of backup philosophy. But it is a mythologized state, a still frame pulled from a moving picture. What about the subtle, unlogged configuration change made by a departing colleague six months ago that’s essential for a critical but rarely used feature? What about the complex, undocumented dependency on an external service that was mocked in the test environment but has since changed its API? A restore test can verify that the bits are present and the services start, but it cannot verify that the restored system can truly perform its intended function in the current world. Our reliance on the backup as a single source of truth makes us neglect the living, breathing institutional knowledge required to actually *use* it.

This is the counterintuitive core of the argument: by making our backups too reliable, too automated, and too trusted, we inadvertently engineer the fragility of our own response. A team that has never truly faced the chaos of a partial or ambiguous restore is a team that will panic when the pristine process inevitably hits a novel snag. The muscle memory for creative problem-solving, for reading the tea leaves of incomplete logs, for stitching together a solution from fragments—that muscle weakens from disuse.

Perhaps, then, the health of our operational resilience isn't measured by the flawless execution of a restore, but by our practiced ability to navigate a flawed one. We should occasionally—deliberately—introduce small, controlled glitches into our recovery drills. Restore from a backup that's missing a key file. Simulate a scenario where the primary and secondary backups are both compromised and you must work from a weeks-old tertiary copy. The goal is not to test the system's perfection, but to test our own imperfection and adaptability. The reliable system is not the one that never breaks, but the one that, when broken, is surrounded by people who know how to fix it, even when the manual is lost.

Notes & further reading

A few pages I came back to while writing this: