The Unwritten History: Manual Playbooks vs. Immutable Procedures
There is a sacred text in our operations realm, often called the ‘playbook’. In its ideal form, it is a meticulously maintained document, a chronicle of institutional wisdom. It’s the step-by-step guide for when the database refuses connections, when the web nodes start reporting 500 errors at 3 AM, the go-to scripture for the engineer on call. It lives in a wiki, its version history a palimpsest of past crises and their solutions. It is, in theory, a map for the firefight. But, like any map drawn from memory, its accuracy degrades with time, and its utility is entirely dependent on the skill and presence of the map-reader.
This contrasts sharply with another philosophy, one that mistrusts the manual entirely. In this world, the procedure is not a document but an immutable script. There is no ‘playbook’ for recovering the primary database node; there is, instead, a single, audited, version-controlled script called `promote-replica.sh`. The knowledge isn't documented; it is encoded. The senior engineer's tacit understanding of edge cases and dependency checks is translated into conditional logic and automated verification steps. The procedure isn't something you read and follow; it's something you execute, and it either succeeds or fails with a clear exit code. The human role shifts from interpreter to overseer.
The Human in the Loop
The playbook, for all its potential fragility, enforces a crucial human-in-the-loop mechanism. Following a checklist, even a slightly outdated one, forces a moment of cognition. It creates an opportunity for the operator to notice something amiss that the script would blindly ignore—a strange log line, an unusual metric, a gut feeling born of experience that something else is fundamentally wrong. The manual process is a conversation with the system, however slow and error-prone that conversation might be. It is the embodiment of the 'why' behind each action.
The immutable procedure, by contrast, prizes consistency and speed above all else. It eliminates the variance of human error, of skipped steps, of misread instructions. It is ruthlessly efficient, but it is also silent. When it fails, it often fails spectacularly and opaquely, because the context for its actions is hidden within its code. You traded the ambiguity of a human interpreter for the inscrutability of a machine that does exactly what it was told, for better or worse. The ‘why’ is buried in a commit message from six months ago.
Both approaches are, in their own way, acts of faith. The playbook trusts that human judgment, guided by written wisdom, will navigate the unique chaos of a real-world incident. The immutable procedure trusts that we can codify that judgment so perfectly that the human becomes a redundant, slower component in the recovery path. The tragedy of the playbook is that it becomes obsolete the moment it’s written, a snapshot of a system that is constantly evolving. The tragedy of the immutable procedure is that it can only handle the failures we were clever enough to anticipate.
I find myself straddling these two worlds. For complex, rare, and high-stakes procedures—the kind that involve potential data loss—I still want the deliberative pace of a playbook, even with its risks. It’s the difference between an autopilot that can land a plane in a calm simulator and a pilot who can reason their way through a sudden crosswind. But for the repetitive, well-trodden paths of failure, the scripts are invaluable. They are the tireless assistants that handle the boring stuff, freeing us to focus on the genuinely novel problems. The real art of ops, then, isn't choosing one over the other, but knowing which kind of history you need to write for which kind of tomorrow.
Notes & further reading
A few pages I came back to while writing this: