The Unnoticed Erosion: On the Slow Decay of Your Restore Procedure
We talk about backups with a kind of religious fervor. We meticulously script them, we monitor their success, we send ourselves triumphant green-check alerts when they complete. We store them across continents, on immutable storage, in triplicate. We have, we tell ourselves, achieved backup nirvana. The data is safe. But this is where the great, quiet lie of operations begins. We have conflated the act of backing up with the ability to restore, and they are not the same thing.
A backup is a snapshot of bits at a moment in time. A restore is a complex procedure that happens in a real, and likely broken, environment. And procedures, unlike bits, are subject to entropy. The script you wrote two years ago to mount the backup volume assumes an older kernel version. The authentication method for the backup service changed six months ago, and while the new backup script accounts for it, the restore documentation gathering dust in the wiki does not. A critical piece of middleware was upgraded, and the restored application, from a time before that upgrade, will fail to start, complaining about a missing library you’d forgotten was ever there.
This is the unnoticed erosion. It’s the slow, silent drift between the perfectly preserved fossil of your backup and the living, changing ecosystem of your production system. Your backup is a time capsule, but you’ve lost the instructions on how to open it without it crumbling to dust. We are excellent archivists and terrible archaeologists.
The Test of Time
The received wisdom is to test backups. But we must critique the shallow interpretation of this idea. Running a checksum on a file to prove it’s not corrupt is not a test of a restore. It is a test of file integrity, and that is a dangerously low bar. A true test is a fire drill. It is declaring a quiet Tuesday afternoon a disaster and demanding the team bring the entire service back online from scratch, using only the backups and the documented procedure. No shortcuts, no sneaky peeks at the live configuration.
The outcome of this drill is never a binary pass or fail. It is a measure of the friction. How many undocumented steps were there? How much frantic searching through old Slack channels occurred? How many “oh, right, we have to manually create that directory first” moments were there? This friction is the measurable decay of your restore procedure. It is the grit in the gears that will grind your recovery time objective to a halt when the lights are actually out and the pressure is on.
Our obsession must shift from the green checkmark of a successful backup to the honest, bruising narrative of a failed restore drill. The logs from that drill are the most valuable ones you’ll ever read. They are the map that shows where your real system has drifted from its archived shadow, and they are the only thing that can guide you back to true safety.
Notes & further reading
A few pages I came back to while writing this: