The Silent Partner: Building a Watchdog for Your Cron Jobs
We trust them with our most critical routines: the database backup, the log rotation, the cache purge. They are the cron jobs, the silent workers of our systems, toiling away in the pre-dawn hours. But who watches the watchers? Who ensures the backup script didn't hit a full disk and fail silently, or that the API sync didn't choke on a malformed response? We set them and forget them, a dangerous act of faith in an imperfect world.
The answer isn't more complex orchestration. It's a simpler, more fundamental practice: the cron watchdog. This is a technique, not a tool. It’s the principle of having one cron job check on the well-being of another. It is the simplest form of operational empathy, a way for your systems to care for their own.
Here is the concrete, practical technique. For every critical cron job you have, create a companion. This companion’s sole duty is to look for the fresh evidence of its partner’s successful completion. If your backup job writes a file to /backups/daily/db-backup-$(date +\%Y\%m\%d).sql.gz, then your watchdog job, scheduled an hour after the backup is meant to finish, runs a simple check: does that file exist? Is it larger than zero bytes? Perhaps it checks the logs for a specific “success” string.
The script itself is often just a few lines. An if statement. A test -f command. Its failure condition is where the magic happens. If the check fails, it must sound an alarm. This could be as straightforward as a loud curl command to a service like Healthchecks.io, or a simple email sent via mailx. The goal isn’t to fix the problem automatically—that way lies more complexity and potential failure. The goal is to fail loudly, to break the silence that surrounds a nocturnal failure and bring it into the light of day where a human can address it.
This practice transforms your relationship with scheduled tasks. They are no longer ghosts in the machine, hopes set adrift in a sea of crontab entries. They become accountable. Each important process gains a silent partner, a dedicated sentry whose only job is to notice its absence. It is a small, boring, and profoundly reliable piece of technology. It asks a simple question of your systems over and over: is everything where it should be? And when the answer is no, it doesn’t try to be a hero. It just turns and tells you.
Notes & further reading
A few pages I came back to while writing this: