The Forgotten Ledger: On the Necessity of the Manual Check
We build systems to watch our systems. We set alerts for disk space, for memory, for process health. We trust these automated sentinels to sound the alarm, and we trust ourselves to respond. It’s a comfortable, modern arrangement. But comfort, in our line of work, is a precursor to surprise.
I propose a small, quiet rebellion against total automation. I suggest you keep a ledger. Not a digital one, not a spreadsheet or a database, but a physical notebook. A simple, unassuming book with blank pages. And once a week, you perform a manual check.
The ritual is simple. You open the ledger to a fresh page and write the date at the top. Then, you begin the walk. You SSH into your primary server and run `df -h`. You don’t wait for a script to parse the output; you read it. You see that the `/var` partition is at 67% capacity. You note it down in your book: "Primary Server - /var - 67%". You check the last ten lines of your critical application log, not because an error triggered a PagerDuty alert, but to see the rhythm of its operation. You note any unusual warnings.
You move to the backup server. You don’t just check if the last backup job exited with a `0` status. You actually list the most recent backup file, note its size and timestamp in your ledger, and verify it exists in your off-site storage target. You write it all down.
The Unseen Benefit
The immediate benefit is obvious: you catch the slow creep of a full disk that hasn’t yet tripped an 80% threshold. You spot the backup that’s been silently failing for three days. But the deeper value is in the act itself.
The physical act of writing engages a different part of your brain than skimming a dashboard. It forces a pace of deliberate observation. You are not reacting to a screaming alert; you are conducting a survey. This slow, manual process builds a profound, intimate familiarity with the baseline hum of your systems. You start to know what ‘normal’ looks and feels like, which makes the subtle anomalies—the ones automation often misses—stand out like a wrong note in a familiar song.
This ledger becomes your ship’s logbook. It’s a persistent record that exists outside of the cycle of log rotation and monitoring tool upgrades. In a crisis, when your fancy dashboards are failing to load, you can flip through its pages and see the patterns that led to the event. It is the ultimate fallback, the one system that never goes down.
In a world obsessed with hands-off operation, the hands-on check is a radical act of care. It is the keeper’s walk around the lighthouse, the gardener’s daily inspection of the soil. It is boring, profoundly unsexy, and one of the most reliable techniques I have ever employed. Start your ledger this week. You might be surprised by what you find, simply because you were finally looking.
Notes & further reading
A few pages I came back to while writing this: