The Fallacy of the Full-Machine Backup
For decades, the cardinal rule of system administration has been a simple one: take regular, full-machine backups. The image, the snapshot, the complete byte-for-byte copy of a server’s drive. It’s the ultimate safety net, the magic ‘undo’ button for catastrophic failure. We are taught to venerate this process, to build intricate cron jobs and shipping protocols to ensure that if the building burns down, we can spin up a perfect digital twin in the cloud. But I want to pose a dangerous question: what if our devotion to the full-machine backup is making us weaker, more fragile operators?
The Crutch of Completeness
The problem with a perfect backup is that it encourages imperfection in our daily practice. When the ‘golden image’ is always waiting in the wings, the incentive to truly understand the intricate, often messy, dependencies of our services diminishes. Why meticulously document the configuration of that obscure daemon when you can just `dd` the whole drive and call it a day? The full backup becomes a crutch, allowing operational debt to accumulate silently. We stop treating our systems as living, understandable entities and start seeing them as monolithic black boxes that we can simply replace, not repair.
Worse still, the restoration of a full-machine backup is often a sledgehammer solution. It’s an all-or-nothing event that rarely accounts for the subtle state of a running system. That restored database might be consistent on disk, but what about the transactions that were in flight? What about the stateful connections that were severed? The blunt force of a full restore can itself be a form of data loss, obliterating the nuanced present to revert to a clumsy, if consistent, past.
Instead, I advocate for a more surgical approach. Focus on backing up the data, not the machine. Treat your infrastructure as ephemeral, its configuration as code. The goal should be to be able to rebuild the *service* from scratch using automation, pulling in only the precious, irreplaceable data from your backups. This forces a discipline that the full-image backup avoids. You are compelled to understand how your service is built, how its parts interconnect, and what constitutes its true, vital state.
A backup of a PostgreSQL database, an export of the key-value store, a copy of the user-uploaded assets—these are the things that matter. The operating system, the package manager’s cache, the temporary files? They are disposable. By focusing our efforts here, we create a system that is not only more resilient but also more transparent and easier to reason about. When disaster strikes, we are not performing a mystical resurrection of a dead machine; we are calmly orchestrating the rebirth of a service on a fresh, clean slate. It shifts the paradigm from passive, fearful archiving to active, confident engineering. The safety net remains, but it no longer encourages a fall.
Notes & further reading
A few pages I came back to while writing this:
- El Paso, TX
- The Unblinking Eye of the Manchester Mark 1
- Miramar, FL
- The One-Handed Boot: On the Persistence of Old Errors
- a useful directory
- The Hum of the Spindle: On the Constancy of the Unseen Drive
- a practical rundown
- a local resource
- a regional guide
- one area's overview
- a helpful reference
- a place-by-place guide
- a nearby resource