Why Are My Backups So Quiet?

You've seen the emails. The nightly cron job runs, a script executes, and a little notification lands in your inbox: "Backup successful." It's a ritual. It's boring. It's, well, quiet. You tick the box in your mental checklist and move on. For months, or even years, this quiet hum is the sound of a job well done. But then, a small, nagging thought creeps in: is this silence a sign of peace, or is it the calm before a storm we’re completely unprepared for?

This question hit me last week when a colleague needed to restore a single, accidentally deleted file from our file store. The backup system, which had been faithfully reporting success every night, was our obvious salvation. We navigated to the backup directory, browsed to the correct date, and found… nothing. The file wasn't there. A wave of cold dread washed over me. Was the entire backup compromised? After a frantic hour of digging, we discovered a subtle bug in the pre-backup script that had failed to include a specific subdirectory for the past eight months. The main database dumps were safe, but a whole class of user-uploaded assets was silently missing. The system was never "broken" enough to send a failure alert; it was just… incomplete. And it never made a sound.

This incident revealed the central flaw in our quiet backup process: we were only monitoring for loud failures. We had alerts for when the backup job crashed, but none for when it succeeded in a degraded state. A successful backup notification tells you the *process* finished, but it rarely tells you anything about the *content* of the backup. Did it capture all the files? Is the resulting archive corrupt? Is it the right size compared to yesterday's? We were celebrating the launch of the rocket without checking if the payload had actually deployed.

The Virtue of a Noisy Success

The solution isn't to crave failure; it's to make success a little noisier. We need to move beyond the binary pass/fail and introduce the concept of a "verified success." This means building a simple, automated post-backup sanity check.

For us, this took two forms. First, we added a step that performs a test extraction of a single, known file from the newly created backup archive. If that extraction fails, the entire job is marked as a failure, even if the `tar` or `rsync` command returned a happy exit code. Second, we started logging the size of the backup archive. A script now compares today's size to a rolling average of the last seven days. If the size deviates by more than a certain percentage (either a suspiciously small drop or an impossibly large spike), it triggers a warning. It’s not perfect, but it’s a canary in the coal mine for major data loss or inclusion of unintended, massive temporary files.

This new layer of verification has broken the unsettling silence. Our backups are no longer quiet. They now whisper little details about their health: "Backup successful. 1.2GB archive. Verified test extraction." It’s a small, boring piece of log output, but it’s a whisper I've learned to trust far more than the deafening silence of a blanket "success." Because in the world of operations, the loudest warning is often the one that was never uttered.

Notes & further reading

A few pages I came back to while writing this: