The Quarry and the Spire: On the Burden of the Immutable

There is a gospel that has been preached in the halls of modern operations for some time now, and its central tenet is immutability. We are told to treat our infrastructure as cattle, not pets. Servers, we are assured, should be ephemeral, spun up from pristine, version-controlled images and cast aside without a second thought at the first sign of trouble. The ‘golden image’ is held up as the ideal: a perfect, unchanging template of reliability, stamped out on demand. This philosophy, born from the need to scale and to tame configuration drift, has the sheen of pure logic. But I’ve come to see a quiet tyranny in this perfection, a burden we may not have fully accounted for.

Consider the metaphor of the quarry and the spire. The immutable infrastructure is the quarry. It is a source of perfect, uniform blocks. Each block is identical, flawless, and can be slotted into any part of the structure with the confidence that it will fit. The system that builds and deploys from this quarry is the architect, capable of erecting a functional spire with breathtaking speed and precision. When a block cracks, it is simply discarded and a new one is hauled up to take its place. This is the promise: endless, faultless replicability.

But a spire built only of quarried stone is brittle. It has no give. A subtle shift in the ground, a wind of unexpected force, and the entire edifice is at risk because there is no inherent flexibility, no woven strength from varied materials. More critically, the focus becomes entirely on the perfection of the block, not the character of the spire. The system’s personality—the small, human-scale adjustments, the quirky but effective workarounds that emerge from lived experience—are all bulldozed in the name of uniformity.

This is the burden: the erosion of situated knowledge. When we cannot log into a ‘cattle’ server to run a one-off diagnostic command, when we are forbidden from making a small, fleeting edit to a config file to test a hypothesis, we lose a vital feedback loop. The machine becomes a black box that only speaks the language of its blueprint. We become administrators of a template, not caretakers of a living system. The logs might tell us a block has failed, but they cannot tell us why it *felt* strange an hour before it cracked. That intuition, born from poking and prodding at the unique, ‘pet-like’ machine, is sacrificed.

I am not arguing for a return to the chaos of snowflake servers, manually configured and tended to with sentimental attachment. The benefits of automation and consistency are undeniable. But I question the received wisdom that total immutability is the only path to reliability. Perhaps true resilience lies in a more nuanced approach. Maybe we need designated ‘observer’ nodes, allowed to exist as semi-pets within the cattle herd, instruments for gaining that deeper, more intuitive understanding. Or perhaps we need to build tooling that allows for safe, audited, and transient interventions—a way to ‘whisper’ to the machine without breaking its immutable contract.

Reliability is not just about having a perfect, endless supply of replacement parts. It is also about understanding the stresses that cause those parts to fail in the first place. By worshipping at the altar of the immutable quarry, we risk building a spire that stands tall and straight, yet is hollow, devoid of the learned wisdom that comes from getting our hands dirty with the stone itself. We trade the messiness of understanding for the clean silence of the blueprint, and in doing so, we may be building a quieter, more profound kind of failure.

Notes & further reading

A few pages I came back to while writing this: