The Virtue of the Unverified Backup
In the cathedral of operations, one commandment is chiseled deeper than any other: thou shalt verify thy backups. It is the unassailable, unquestioned pillar of our practice. We are taught to schedule regular restore tests, to compare checksums, to prove, beyond a shadow of a doubt, that our digital lifeboats are seaworthy. To suggest otherwise is heresy. And yet, I am here to whisper a quiet heresy: sometimes, the most reliable backup is the one you never verify.
This isn’t an argument for negligence. It’s an argument for recognizing the true, heavy cost of verification. That cost isn’t just the compute cycles for a checksum or the bandwidth for a file transfer. The profound cost is in the system complexity, the brittle automation, and the psychological burden of a process that must, by its nature, touch and interrogate the very data you’re trying to keep pristine and at arm’s length.
Consider the backup system that is so elegantly simple it’s barely a system at all: a `rsync` or `borg` command writing encrypted blobs to an object store in a different region, run by a cron job that has zero permissions to read what it just wrote. It’s a one-way street. To verify it, you must build a parallel infrastructure—a restoration environment, a decryption key holder, a validation script—that reintroduces the very access vectors and failure modes you designed the simplicity to avoid. Your ‘verification pipeline’ becomes a second, more complicated system you now have to maintain and trust.
The Anxiety of the Green Checkmark
Worse is the psychological trap. A verified backup creates a false sense of security, a green checkmark that says “You Are Safe.” But it only proves the backup was good at the moment of verification. The very next write could be corrupt. We sleep soundly, trusting the ceremony of verification, while the actual, continuous stream of backups goes unexamined. We’ve traded constant, implicit risk for a periodic, explicit ceremony that breeds complacency.
The unverified backup, by contrast, maintains a healthy tension. It acknowledges the act of backup as an act of faith, a humble admission that we cannot control everything. Its reliability is built on a different foundation: extreme simplicity of operation, immutability of the backup target, and a clear, singular focus on the write path. You monitor for failures in the process, not the product. An alert fires if the job doesn’t run, not if a byte is out of place. You invest your effort in making the one-way transaction as bulletproof as possible, and you accept that the ultimate test will be the disaster itself.
This is not for every dataset. Your customer transactions need verification. But what about your static asset bucket, your CI cache, your build artifacts? We reflexively apply the verification dogma to mountains of data that are, in a true disaster, merely inconvenient to lose, not catastrophic. For these, the complexity tax of verification is a poor investment. The unverified backup is a conscious trade: we exchange the certainty of a test for the greater certainty of a system so simple it has almost nothing left to break. It is the acknowledgement that sometimes, the safest hand is the one that only knows how to let go.
Notes & further reading
A few pages I came back to while writing this: