The Unseen Cost of the Perfect Snapshot
We are taught, from the very first moment we touch a server, that backups are sacred. The common wisdom is absolute: more frequent backups are inherently better. The goal is to minimize data loss, to shrink the Recovery Point Objective (RPO) down to mere seconds. We chase this ideal with relentless snapshots, continuous data protection, and complex replication topologies. We strive for the perfect, lossless snapshot of our systems, believing it to be the pinnacle of operational maturity. But I want to challenge that. I want to argue that this pursuit of perfection can, paradoxically, make our systems more fragile and our recoveries less reliable.
The issue isn't the act of backing up, but the hidden weight of the restore. A backup is not a success until the data is live and serving requests again. Every incremental byte we capture, every micro-transaction we journal, adds immense complexity to the restoration process. A system restoring from a snapshot that is only seconds old is often rebuilding a house of cards from a photograph taken during a windstorm. It's attempting to reconstitute a profoundly complex, often inconsistent, state. The recovery becomes a high-wire act, as likely to fail from the sheer intricacy of the reassembly as it is to succeed.
The Beauty of the Clean Break
There is an elegance, a brutal simplicity, in recovering from a slightly older, colder backup. Consider a system that backs up fully, but only once an hour. In the event of a catastrophe, you lose up to 59 minutes of data. This sounds like a nightmare to the RPO-obsessed. But what you gain is a known, stable, and tested state. That backup was taken at a point of relative calm. The restoration process is a known quantity; it's a procedure you've drilled countless times because the variables are fewer. You are rebuilding from a solid foundation, not trying to catch a falling knife.
This approach forces a healthier architectural discipline. Knowing you have a meaningful gap to fill encourages the design of systems that can tolerate data loss gracefully. It pushes you to build with idempotency and replayability in mind. Instead of a frantic scramble to restore a perfect-but-delicate state, you have a clear, methodical process: restore the last known good state, and then replay idempotent events or regenerate state from the intervening period. The recovery is slower to come fully online, but it is far more likely to come online correctly and stay online.
Our obsession with the RPO number has blinded us to the RTO—the actual time to recovery. A perfect snapshot that takes six hours to restore and verify is worse than a slightly imperfect one that takes ten minutes. Reliability isn't just about the quantity of data saved; it's about the predictability of the process to bring it all back. Sometimes, the most reliable backup strategy is the one that gives you just enough, not everything. It’s the counterintuitive peace that comes from knowing your system can withstand a clean break and be confidently rebuilt, rather than fearing the intricate failure of a perfect restoration.
Notes & further reading
A few pages I came back to while writing this: