The Necessary Lie: On the Deception of Immutable Infrastructure

We preach immutability like a gospel. The doctrine is clear: never modify a running system. If a change is needed, you build a new, pristine image from a known state, deploy it, and burn the old one to the ground. It’s clean, it’s auditable, it’s versioned. It is, in the parlance of our times, the right way to do things. But in our small server rooms and on our modest VPSs, this ideal is often a carefully maintained deception—a necessary lie we tell ourselves to feel secure.

The truth is, absolute immutability is a fantasy for most small operations. The promise hinges on a perfect, automated pipeline from commit to production, a symphony of CI/CD scripts and orchestration tools that hums along without a hitch. But what happens when the music stops? When a zero-day vulnerability drops and the official base image in the registry hasn’t been patched yet? Do we wait, exposed, for the pipeline’s custodian to wake up and rebuild the world? Or do we, just this once, ssh in and run `apt-get update`?

We do the latter, of course. We whisper the incantation onto the live system, our hearts pounding with a heretic’s thrill. We document the change in a ticket we swear we’ll reconcile later. We pretend this temporary mutable sin is an anomaly. But it isn't. It’s the pressure release valve that keeps the entire ideology from exploding. This ‘sin’ is often the very thing that keeps the service running.

The Sanctity of the Working State

The purist’s argument is that this one-off change creates drift, a snowflake server that can never be reliably reproduced. But this ignores a more profound truth: a running system is itself a unique state. Its memory contains caches warmed by real traffic, its connections are live arteries to other services, its state is a complex tapestry woven in real-time. The new, ‘immutable’ replacement will start cold and naked, and its performance out of the gate is a gamble.

For a colossal distributed system, this cold start is a calculated risk. For a small service, it can be a silent failure point. The ‘immutable’ deployment might be perfectly identical to its blueprint, but it is not identical to the system it replaced—the system that was actually, tangibly working. Our reverence for the blueprint can blind us to the sanctity of the working state.

This isn’t a call for anarchy. It’s a plea for intellectual honesty. Immutability is a brilliant pattern, a powerful guiding star. But we must admit it is often an aspiration, not a reality. The real wisdom lies not in dogmatically refusing to touch a live system, but in knowing precisely when that rule must be broken to serve a higher goal: continuity itself. Our infrastructure isn't immutable; it's resilient. And sometimes, resilience looks a lot like a carefully applied, well-documented, temporary patch.

Notes & further reading

A few pages I came back to while writing this: