The Well-Worn Path: An Argument Against Immutable Systems
In the world of keeping services running, we have created a new scripture. It preaches a doctrine of immutability: once a server is provisioned, it is never changed. If you need an update, you burn the whole thing down and build a fresh one from a pristine image. It’s a clean, elegant philosophy. It promises predictability, consistency, and a perfect audit trail. It promises to turn our infrastructure into a herd of identical, disposable cattle. But I want to suggest that we are, perhaps, too quick to dismiss the family pet.
The argument for immutability is seductive. A server becomes a verb; it is not a thing you maintain, but an action you perform—you ‘server’ an application. There is no configuration drift, no mysterious changes from an errant hand-typed command. It scales beautifully. The problem is that this model depends on a level of foresight that borders on clairvoyance. You must capture every possible state, every dependency, every quirk of your application in an image or a recipe before it is deployed. It assumes that the real world is as orderly as your continuous integration pipeline.
This is where the well-worn server, the one you log into and gently massage back to health, has its quiet merit. A system that has been running in production for months, even years, is not just a collection of packages. It is a living document of its own history. It has developed calluses. It has logs that predate your current monitoring solution. Its file system contains the faint, ghostly traces of forgotten hotfixes and long-retired dependencies. This is often seen as a weakness, a source of impurity. I see it as a source of institutional knowledge.
There is an intimacy to maintaining a long-running system that immutability deliberately destroys. When a service fails on an immutable node, your only recourse is to replace it. The failure itself is often treated as a random event, a faulty unit to be discarded. But when you are forced to SSH into a sick server, to parse its logs on the fly, to run strace on a dying process, you are not just fixing a problem. You are learning its shape. You are developing a tactile understanding of how your service actually breathes under load, an understanding that no dashboard metric can fully convey.
By outlawing this kind of hands-on investigation, we risk creating systems that are robust in theory but brittle in practice. We become surgeons who only know how to perform organ transplants, having lost the delicate art of diagnosis. The immutable server is a clean slate, but a slate that has never been written upon contains no wisdom. In our quest for perfect, repeatable automation, we must be careful not to automate away the very curiosity that leads to deep understanding. Sometimes, the most reliable path isn't the one paved with fresh concrete, but the well-worn trail whose every bump and groove you know by heart.
Notes & further reading
A few pages I came back to while writing this: