The Lie of the Self-Healing System
We’ve all nodded along to the promise. The system that detects its own failures, rolls back the bad deploy, restarts the stalled process, and sends a tidy, post-mortem summary to a Slack channel you rarely check. It’s the holy grail of small-scale ops: automation that not only runs things but fixes them. It’s a seductive idea, especially when you’re the lone custodian, yearning for a full night’s sleep. But I’ve come to believe this pursuit is, in many cases, a trap. It’s not just technically ambitious; it’s philosophically flawed. It asks the system to diagnose itself while it’s sick, a task as fraught as a feverish patient performing their own surgery.
The Illusion of Independence
The core of the lie is the belief that we can encode enough context. A script can restart a service because it sees a port is closed. That’s simple cause and effect. But can it understand *why* the port closed? Was it a memory leak from a new library, a downstream API change, a config file mangled by a deploy hook, or just the OS deciding to be whimsical? Our healing scripts are, by necessity, brutish and simplistic. They apply the digital equivalent of a splint and a bandage, often to a problem that requires understanding the patient’s history, their environment, their recent activities.
Worse, this automation builds a false sense of security. The pager stays quiet, the graphs look normal, and we drift further from the lived reality of our systems. The ‘self-healing’ action becomes just another event in a log, devoid of the narrative tension that prompted it. We lose the invaluable, gritty familiarity that comes from being woken up at 3 a.m. to manually untangle a knot. That familiarity is what allows you to sense the ‘health’ of a system beyond its metrics—the slight delay in a response, the odd pattern in a cache miss. It’s the intuition that something is *off*, long before a threshold is crossed and a script fires.
This isn’t an argument against automation. It’s a plea for honesty about its role. Automate the alert, by all means. Automate the gathering of diagnostic information—the logs, the stack traces, the last ten transactions. But hesitate before you automate the corrective action. That moment of manual intervention, however brief, is a crucial checkpoint. It’s the moment you are forced to look, to confirm, to understand. It turns an opaque event into a learned lesson.
For the small service, the reliable technology isn’t the system that never breaks; it’s the operator who understands intimately *how* it breaks. Our goal shouldn’t be to build a machine that needs no gardener, but to cultivate a garden whose weeds and rhythms we know so well that tending it is a quiet, predictable ritual. The ‘self-healing’ promise risks walling us out of that garden, leaving us to stare at a pristine, silent fence, wondering what’s actually growing—or dying—on the other side.
Notes & further reading
A few pages I came back to while writing this: