The Silent Sentinel: A Simple Script to Verify Your Backups Are Alive
We talk about backups with a kind of religious fervor. We preach the 3-2-1 rule, we meticulously schedule our `rsync` or `borg` runs, and we rest easy knowing that our data is safely tucked away in some distant corner of a cloud bucket. But faith, as it turns out, is a poor substitute for verification. A backup you never test is merely a prayer. It’s a promise you’ve made to your future self, with no guarantee it will be kept.
The most common point of failure isn't the backup process itself; it's the silent, creeping rot of assumption. The script runs, it returns a success code, and we move on. But what if the destination mount point failed? What if the credentials silently expired? What if the backup completed but the resulting archive is corrupt, a hollow shell of bits that looks whole but crumbles at the first touch of a restore command? We don’t discover these failures until the moment of absolute need, which is the worst possible time for a revelation.
The Heartbeat Check for Data
The solution isn't complex, but it is deliberate. Beyond the backup script, you need a sentinel. A separate, independent process whose sole purpose is to prove that the backup not only ran, but produced a living, breathing file. My sentinel of choice is a small script that runs shortly after the backup completes. It doesn't check the entire archive—that would be too costly. Instead, it performs a single, vital test: it attempts to read the last few bytes of the backup file.
Here’s the core of it, a simple Bash one-liner that packs a punch of assurance:
tail -c 1024 /path/to/backup/file.tar.gz | wc -c | grep -q 1024
This command grabs the final kilobyte of the backup file. The pipe to `wc -c` counts the bytes it successfully read. The `grep -q 1024` then silently checks that it indeed read 1024 bytes. If any part of this chain fails—if the file doesn't exist, if it’s unreadable, or if it ends prematurely—the command returns a non-zero exit status. This failure can then trigger an alert, screaming into your monitoring system that the prayer has gone unanswered.
You wrap this in a script that runs an hour after your backup, and you have a canary in the coal mine. It’s a tiny, almost negligible consumption of resources for an immense return: the peace of mind that comes from knowing your safety net isn’t made of wishes. It’s a simple, boring check. And like all boring technology, its true beauty is found not in its complexity, but in its steadfast reliability.
Notes & further reading
A few pages I came back to while writing this: