The Whisper in the Attic: A Guide to Passive Heartbeat Monitoring

We spend a lot of time teaching our systems to scream. We wire them up with elaborate alerting rigs, setting thresholds and configuring pager duty rotations, all so they can shriek the moment something goes wrong. This is necessary, of course. But there’s another sound a healthy system makes, one we often forget to listen for: its heartbeat.

A heartbeat is the quiet, rhythmic pulse of a functioning service. It’s the cron job that completes, the script that logs its success, the daily backup that writes its final timestamp. We expect these things to happen, and when they don’t, their absence is a silent, growing void. The problem with silence is that it’s easy to miss until it’s too late. We need a way to notice the nothing.

This is where passive heartbeat monitoring comes in. It’s a simple, boring, and profoundly reliable technique. Instead of asking a service "are you okay?" (active checking), you simply listen for it to whisper "I did my thing" and alert only if that whisper stops. The implementation is beautifully straightforward.

The Humble `touch` as a Sentinel

Here’s the concrete how-to. Within any script or cron job that runs periodically—your nightly backup, a log rotation script, a data sync process—add a single line at the very end:

touch /var/heartbeats/my-daily-backup

That’s it. This updates the timestamp of a specific file upon every successful run. The magic happens in a separate, simple monitor. Using a tool like `find`, you can check for the absence of this whisper. A cron job running every hour could execute:

find /var/heartbeats -name 'my-daily-backup' -mmin +1440

This command will return a result if the file `my-daily-backup` has not been modified in the last 1,440 minutes (24 hours). If it finds the stale file, it triggers an alert. The logic is inverted: we alert on the presence of stale data, not the absence of a response from an actively probed service.

The elegance of this method is its decoupling. The service itself doesn’t need to know about the monitoring system. It doesn’t require a dedicated monitoring agent, an open port, or complex authentication. It just touches a file, a fundamental, atomic operation it was already built to do. The monitor, in turn, doesn’t need to understand the service’s internal logic; it only understands filesystem timestamps, one of the most reliable primitives we have.

This technique is the attic watchman. It doesn’t bother the household below while they work. It just sits in the dark, listening for the familiar creak of the floorboards that says all is well. And when that sound doesn’t come, it knows, with utter certainty, that something is wrong. It’s a practice that values quiet consistency over noisy interrogation, and in that quietness, finds a deeper kind of reliability.

Notes & further reading

A few pages I came back to while writing this: