The Necessary Rupture: On the Danger of Too Much Stability
Our entire discipline is built on a foundation of fear. Fear of the crash, the outage, the data loss. We build fortresses of redundancy, weave safety nets of backups, and script away every conceivable point of human failure. We chase the holy grail of the ‘five-nines,’ aiming for a state of perfect, uninterrupted service. This is the common wisdom, the unassailable goal. But I want to propose a heretical thought: what if this relentless pursuit of stability is, itself, a profound risk?
We’ve all seen it. The system that has hummed along for years, untouched, unaltered, its original architects long gone. It becomes a black box of perfect, boring reliability. We celebrate its uptime, but we quietly dread the day it finally stutters. Because that system is also a fossil. Its dependencies are ancient, its knowledge siloed, its recovery procedures untested and likely forgotten. Our obsession with never touching it has made it incredibly fragile. The longer the period of perfect stability, the greater the potential energy of the eventual disaster.
The Virtue of the Deliberate Tremor
This is why we must embrace the idea of the necessary rupture. Not a catastrophic failure, but a controlled, deliberate one. We must intentionally introduce small, managed breaks in the system to keep our understanding and our recovery muscles sharp. This is the counterintuitive core: to prevent the big break, you must practice with small ones.
This means occasionally, and on your own terms, killing a primary node during low traffic to prove the failover works as advertised. It means scheduling a drill where you restore a critical database from last week’s backups into a staging environment, not because you have to, but because you can. It means turning off a logging aggregator for an hour to see what gaps appear in your monitoring and who actually notices.
These acts feel like sacrilege. They introduce risk where none existed. But they replace a massive, unknown, and accumulating risk with a series of tiny, known, and managed ones. Each small rupture teaches you something. It exposes a faulty assumption, a stale runbook, a missing permission, a piece of ‘tribal knowledge’ that never made it into the docs. It turns your system from a revered relic into a known quantity.
Perfect, uninterrupted stability is a siren song. It lulls you into a false sense of security, allowing operational debt to accumulate silently in the shadows. True resilience isn’t the absence of failure; it is the proven, practiced, and confident ability to meet failure head-on and recover from it. Sometimes, the safest thing you can do for your system is to break it on purpose.
Notes & further reading
A few pages I came back to while writing this: