The Humble Ledger: How to Keep a Manual Runbook That Actually Gets Used

There’s a quiet truth in operations that we rarely say out loud: the most sophisticated automation sometimes fails you. It’s at 3 a.m., with the dashboard screaming red and the familiar automation scripts returning cryptic errors, that you find yourself adrift. You need to *think*, not just execute. This is the moment for a different tool, one older than any config management system: the manual runbook. Not a sprawling Confluence graveyard, but a single, living document built for the human in the crisis. I call mine the Ledger.

The Ledger’s power is in its constraints and its purpose. It is a text file, plain and simple, named `LEDGER.md` and kept in the root of the service’s repository or config directory. Its sole purpose is to guide a competent but potentially stressed human through restoring service or understanding a novel failure. It is not for onboarding, not for architectural diagrams, and certainly not for listing every conceivable alarm. It is for the firefight.

The Structure of a Single Page

A useful Ledger fits on one screen, scannable in under thirty seconds. It has only three sections. The first is “The Golden Thread” – the one, non-negotiable command or diagnostic that tells you if the core service is fundamentally alive. It might be `curl -f http://localhost:${PORT}/health`, or checking for a specific process footprint. This is your first touchstone.

The second section is “The Three Pillars.” These are the three critical downstream dependencies without which your service is a brick. For a web API, it might be the database, the auth service, and the message queue. For each, you list the single most direct way to check its health from *this* service’s perspective—its connection string, a test query, or a queue depth check. No links; the command or URL is written plainly.

The final and largest section is “The Known Paths.” This is a curated list of past major failures and their concrete, step-by-step resolutions. Each entry has a terse title (“Postgres WAL directory full”), a symptom (“All writes failing, logs show ‘No space left on device’”), and the exact, copy-pastable command sequence that fixed it (`ssh db-primary`, `df -h /var/lib/postgresql`, `sudo systemctl stop postgresql`, etc.). These are not speculative playbooks; they are archaeological records of actual battles won.

The discipline of the Ledger is in its curation. When a novel crisis is resolved, the first post-mortem task is to distill the solution into a new “Known Path” entry. When an entry becomes obsolete—because the infrastructure changed—it is deleted. The file must remain lean, relevant, and trustworthy. It becomes not a bureaucratic requirement, but a personal tool for the on-call engineer, a whisper from their past self to their future, panicked self: “Here. Start here. You’ve seen this before, and this is how we walked out.”

In an age of auto-remediation, the manual Ledger feels almost heretical. Yet, it embodies a deeper reliability: the reliability of human judgment, aided by context. It doesn’t replace automation; it fences the area where automation breaks down, giving your mind the solid ground it needs to build a new solution. It is the fixed point you drill into the cliff face, not to hang your entire weight on, but to give you a moment’s purchase while you find the next handhold.

Notes & further reading

A few pages I came back to while writing this: