The Gentle Art of the Pre-Mortem: Writing the Incident That Hasn't Happened
We spend our days building walls. We configure firewalls, set up alert thresholds, and script health checks—all to keep the barbarians of chaos from the gate. But what if we invited them in for tea, sat them down, and asked them exactly how they plan to breach our defenses? This is the essence of the pre-mortem, a quiet, deliberate practice that turns anxiety into a blueprint for resilience.
Unlike a post-mortem, which dissects an incident after the fact, a pre-mortem is a speculative exercise. You gather your team, present a simple, plausible scenario of a future failure, and ask one question: "How did this happen?" The goal isn't to predict the future with clairvoyant accuracy. It’s to probe the soft, overlooked edges of your system where assumptions have quietly congealed into fact.
The Quiet Ritual of the Hypothetical
Start small. Don’t try to envision the apocalypse. Choose a specific, modest service—perhaps your internal authentication gateway or that cron job that generates nightly reports. In a shared document, write a single, stark headline: "Incident Report: Service X Unavailable for 3 Hours." The date is set for next month.
Now, together, write the narrative. Let the story unfold. Someone will start: "The primary node failed over, but the secondary was already starved for memory from an unrelated process." Another adds: "Our alerts for memory on the secondary were set to warn, not page, so no one was notified." A third might chime in: "The runbook for a failover assumed a healthy secondary, so the steps just made things worse."
You are not assigning blame. You are uncovering single points of failure, alerting gaps, and documentation rot you didn't know was there. The pre-mortem gives your team permission to voice their quiet, nagging doubts without the pressure of a real incident lighting up the room.
The output is not a document to be filed away. It is a direct action list. That memory alert? It’s now critical. That outdated runbook? It’s now a task for next week’s sprint. The assumption that a failover target is always healthy? You’ve just identified a need for a pre-flight check script. You have transformed a vague worry into a concrete, schedulable piece of work. You have woven a new, stronger thread into the fabric of your system, all before a single user ever noticed a thing.
This practice is a humble admission that we cannot foresee every breakage, but we can cultivate a mindset that actively hunts for fragility. It replaces the passive hope that "it won't happen" with the deliberate action of making sure that if it does, you are already prepared. It is the quiet, thoughtful work of listening for the faint cracks in the foundation, long before the storm ever arrives.
Notes & further reading
A few pages I came back to while writing this: