The Unyielding Chain: On the Tyranny of the Immutable Backup
We are taught, from our earliest days tinkering with servers, that backups are sacred. The received wisdom is absolute: a backup must be immutable, a frozen, unchangeable artifact of a moment in time, safe from the meddling hands of a misguided script or a panicked sysadmin. This is the gospel of the immutable backup, and we have built altars to it in cold storage and versioned buckets. But I want to question this orthodoxy. What if, in our quest for perfect preservation, we have forged an unyielding chain that can strangle our ability to recover?
The argument for immutability is, on its face, unassailable. It protects against ransomware, against accidental deletion, against the ‘oops’ that follows a `rm -rf` command issued in the wrong terminal window. It is a digital vault, and a vault must, by definition, be locked. We sleep better knowing our last line of defense is hardened against change.
Yet, this very strength is its insidious weakness. An immutable backup is a perfect record of a perfect moment, including all the imperfections we didn’t notice at the time. It is a snapshot of the latent configuration error that would take the service down again if restored. It is a carbon copy of the corrupted user database, the very problem we are trying to solve. We preserve not just the good state, but the poisoned one, too. In our zeal to create a point-in-time recovery, we have accidentally enshrined our mistakes, making them just as permanent and ‘recoverable’ as our successes.
This creates a paradox of recovery. The act of restoration becomes not a simple reversion, but a delicate archaeological dig. We must restore the immutable backup and then immediately perform surgery upon it to excise the flaw that necessitated the restore in the first place. We are forced to fix the same problem twice: once in the present and once in the past we’ve just resurrected. The immutable backup doesn’t solve our crisis; it merely gives us a sterile environment in which to re-live it.
This is not a call to abandon immutability, but to challenge its primacy. Perhaps the ideal system is not a single immutable monolith, but a layered approach. A recent, mutable ‘working’ backup that can be sanitized and corrected, followed by a deeper, truly immutable archive for catastrophic, ‘burn-it-all-down’ scenarios. The goal should not be rigid immutability for its own sake, but intelligent recoverability. Our highest duty is not to the perfect preservation of a byte, but to the swift and sane restoration of a service. Sometimes, the most reliable backup is not the one that cannot be changed, but the one we are allowed to fix.
Notes & further reading
A few pages I came back to while writing this: