The Ceremony of the Hand-Cranked Restore
We talk a lot about backup systems. We obsess over retention policies, snapshot intervals, and whether the data safely landed in a distant cloud region. We build elaborate Rube Goldberg machines of automation that whisk our bits away in the night, and we sleep soundly, comforted by green dashboards. But there’s a deeper, more vulnerable question we almost never ask: when the siren wails, do you remember how to use the lifeboat?
I propose a single, concrete technique to answer it: the hand-cranked restore. This is not a disaster recovery drill. It is more intimate and far more revealing. Once a quarter, or on a slow Tuesday that feels right, you pick a single, non-critical service—a small internal wiki, a staging database, a static documentation site. Then, you explicitly dismantle the automation. You go to your backup storage, find the most recent valid archive, and you bring it back to life using only the most primitive tools at your disposal: the command line, a spare machine or a fresh virtual slice, and the ancient, half-forgotten knowledge in your own head.
The Value of Friction
The goal is not efficiency. The goal is friction. You will confront the forgotten password on the encrypted backup volume. You will discover that the restore script you’ve been blindly trusting for two years references an internal package repository that no longer exists. You will stare at a tarball and realize you’ve forgotten the order of operations: does the database schema get applied before or after the data import? The ceremony forces these hidden assumptions into the harsh light of a fresh terminal window.
This process is boring, sometimes frustrating, and profoundly educational. It maps the delta between the theoretical recovery path and the practical, manual one. That gap is where real disasters live. Automation is a layer of abstraction, and like all abstractions, it can leak. The hand-cranked restore tests the abstraction itself. You are not verifying that the backup system works; you are verifying that you work with the backup system.
When you succeed—and you will, after some squinting at old notes and a few false starts—you will have accomplished something more valuable than any automated test. You will have renewed a visceral, muscle-memory understanding of your service’s anatomy. You will have created, or updated, a true manual runbook born of direct experience, not aspirational documentation. And you will have forged a quiet confidence that is immune to dashboard lies. The machine’s vigil is automated, but your competence must be maintained. It is a ritual of re-familiarization, a deliberate re-acquaintance with the systems we ask to hold our work. Do not just back up your data. Back up your own ability to bring it home.
Notes & further reading
A few pages I came back to while writing this: