The Unwritten Recipe: On the Myth of Self-Healing Systems

A quiet dogma has taken root in the operations world, whispered in conference halls and written into product brochures: the goal is the self-healing system. The machine that, when poked, prods itself back into shape without a human hand. It is presented as the ultimate state of grace, a nirvana of automation where our work is finally done. But I suspect this is a fantasy, and a potentially dangerous one, because it misunderstands the very nature of healing.

Healing is not merely a mechanical process of restarting a service or scaling a resource. A machine does not heal; it resets. True healing is an act of integration, of learning, of creating a new normal that is stronger for having acknowledged the wound. It is the system administrator who, after a cascading failure, rewrites the deployment playbook to include a new health check. It is the developer who, after a midnight page, adds a new metric to the dashboard that would have revealed the problem hours earlier. These are human acts of synthesis. The system merely provides the alarm; the healing is in the changed human process that follows.

The Silent Regression

To aim for a system that heals itself is to aim for a system that learns nothing. It becomes a black box that occasionally stumbles and then, mysteriously, rights itself. There is no post-mortem, no institutional memory, no hard-won wisdom etched into runbooks or team culture. The "self-healing" system is the ultimate cargo cult, performing rituals whose purpose and meaning are lost even to its creators. When it inevitably encounters a novel failure mode, it will lack the human intuition and adaptability to understand it.

This is not an argument against automation—far from it. Automating the response to a well-understood, routine failure is a mark of maturity. It is the difference between a medic applying a tourniquet and the body clotting a minor cut. One is a deliberate act of applied knowledge; the other is a blind, pre-programmed reflex. We should strive to build systems that are robust, that fail gracefully, and that provide us with the clearest possible signals when something is wrong. But we must not outsource the understanding.

The real, valuable work isn't in building systems that never need us. It's in building systems that tell us, in no uncertain terms, when they do. The healing occurs in the space between the alert and the resolution, where human curiosity meets machine failure. That is where knowledge is written. To seek to erase that space is to seek to stop learning. Our systems shouldn't heal themselves; they should call for the doctor, and in doing so, teach us how to be better physicians.

Notes & further reading

A few pages I came back to while writing this: