A Quiet Rebellion Against Dogmatic Immutability

Immutability has become the unquestioned gospel of modern operations. It’s a powerful idea: you never change a live system; you build a new, pristine artifact from a known state and replace the old one entirely. For sprawling, containerized applications, this is often sound doctrine. It brings consistency, rollback safety, and a clean separation between build and runtime. But in the quiet corners of our infrastructure, where small, persistent services do their unglamorous work, I’ve watched this dogma calcify into a rigidity that sometimes does more harm than good.

I’m speaking of the single-purpose virtual machine running a legacy data transformer, the raspberry pi logging environmental controls, the small database host that’s been purring for years. The immutable mantra, applied here, insists we treat these systems as cattle to be slaughtered and reborn from a master image for every tweak. Yet, the ceremony required—updating the golden image, cycling the instance, re-syncing the persistent state—can be a hundred times more complex than the change itself. We’ve all done it: spent an afternoon orchestrating the ‘proper’ immutable deployment of a two-line configuration fix that could have been applied with `ssh` and `vi` in sixty seconds.

The Tyranny of the Template

This is the tyranny of the template. It assumes all change is equal and must be funneled through the same heavy machinery. It ignores the spectrum of risk. Changing the kernel parameters on a load balancer is not the same as updating a comment in a local cron job. By refusing to distinguish, we create systems where making any change, no matter how trivial, requires the full deployment pipeline. This isn’t resilience; it’s bureaucracy encoded in YAML.

More subtly, dogmatic immutability can erode understanding. When a system is an opaque, black-box artifact that springs fully formed from a CI/CD womb, the operator’s connection to its runtime reality weakens. The intimate knowledge of which config file lives where, how the service binds to its port, how it fails when a disk fills—this ‘sense’ of the system is dulled. We trade the messy, hands-on familiarity for a sterile, repeatable process. For greenfield projects, that’s a fine trade. For the long-lived, small-scale workhorses, it can mean losing the very operational intuition that keeps them reliable.

This isn’t a call to return to the wild west of uncontrolled manual edits. It’s a plea for nuance. Let’s champion ‘mutable cores with immutable shells.’ Keep the deployment artifact and its critical, versioned configuration immutable. But within that running instance, allow for a small, logged, and heavily-audited ‘mutable zone’—a directory for runtime logs, a file for ephemeral tunables, a place where the operator’s touch is permitted, even respected. Document these touches meticulously, stream them to your logs, treat them as exceptions, not the rule.

Reliability doesn’t come from blindly following a single methodology. It comes from aligning your tools and processes with the actual shape of the problem. Sometimes, the most reliable, most boring piece of technology is a stable system that you can understand and adjust with precise, minimal force—not because you’re lazy, but because you’re conserving energy for the battles that truly require the full immutable arsenal. Let’s build systems smart enough to know the difference.

Notes & further reading

A few pages I came back to while writing this: