The Checklist and the Crash: A Ritual for the Unthinkable
It happened on a Tuesday, as these things often do. A frantic message in a normally quiet ops channel: the staging environment was down. Not just a service, but the whole thing—unreachable, unresponsive, a ghost ship on our internal network. The usual reflex is to start poking. `ssh` into the box, check `top`, `df -h`, tail the logs. But this time was different. This time, before anyone could type a single command, a colleague simply said, "Okay, run the checklist."
We have a checklist for this. Not a generic "troubleshooting steps" document lost in a wiki, but a specific, procedural script for a "Complete Host Blackout." It sounds almost melodramatic, but that’s the point. By naming the unthinkable, we make it manageable. The checklist isn’t a panacea; it’s a ritual. Its primary function isn’t to guarantee a solution, but to enforce a pause. It’s the deep breath before the plunge, forcing a shift from panicked reaction to systematic observation.
The Anatomy of a Pre-Mortem
We built our checklist not after a real disaster, but during a quiet afternoon, using a technique I’ve come to call a "pre-mortem." We gathered and asked one question: "Imagine it’s 3 AM. Our main application server is completely dark. No SSH, no monitoring, nothing. What is the absolute first thing we do? And then? And then?" The goal was to document the initial, critical minutes, the ones where adrenaline clouds judgment.
The resulting document is brutally simple. It starts not with commands, but with confirmations. Step one: Confirm the outage is real and not a local network issue by pinging from two separate vantage points. Step two: Check the infrastructure provider’s status page. This sounds obvious, but in a panic, you’d be surprised how many hours can be wasted debugging a problem that exists in someone else’s data center. Step three: Only after these externalities are ruled out do we attempt to access the host console via the provider’s out-of-band management system.
Each step is a single, unambiguous action, followed by an expected outcome. If the expected outcome fails, the checklist branches. For example, if we can get a console but the machine is unresponsive at the OS level, the next step is to grab a memory dump and then initiate a hard reboot. If the console itself is inaccessible, the path leads directly to a ticket with our hosting provider, complete with a pre-written template that includes all the necessary host IDs and initial diagnostic information.
The magic of the checklist is that it externalizes memory. It holds our collective, calm-day wisdom for our panicked-night selves. It prevents the clever, dangerous urge to start hypothesizing about root causes before the basic facts are established. Was it a kernel panic? A failed disk filling the log partition? A misconfigured firewall rule applied earlier in the day? The checklist doesn’t care, not yet. Its sole purpose is to guide us from total blindness to a state of basic awareness, one confirmed fact at a time.
That Tuesday, the checklist worked. We followed the steps. The problem was, in fact, a provider-side network glitch that resolved itself moments after we confirmed it on their status dashboard. The entire "crisis" was over in ten minutes, without a single speculative command executed. No one wasted mental energy. The checklist didn’t fix the problem, but it prevented us from breaking anything else while the system healed itself. It turned a potential late-night drama into a brief, orderly interlude. It’s the most boring piece of emergency technology we own, and I wouldn't run a service without it.
Notes & further reading
A few pages I came back to while writing this: