The Persistent Gardener: On the Art of the Watchdog Script
There’s a particular kind of peace that comes from knowing something is being watched. It’s not the peace of a silent forest, but the peace of a well-tended garden, where the occasional, gentle sound of weeding or watering confirms that things are as they should be. Our small services, the quiet engines that power so much of our digital lives, crave this kind of oversight. They don’t need a firefighter on constant alert; they need a persistent gardener. And for that, the simplest, most reliable tool is often a humble watchdog script.
This isn't about fancy orchestration or complex health-check APIs, at least not at first. This is about a technique so straightforward it feels almost primitive: a script, a cron job, and an exit code. The goal is to create a failsafe for the single most important process on your server, the one whose silence would mean a genuine outage. The logic is beautifully simple: if the process isn’t running, start it. If starting it fails, tell someone. This script is the gardener who, upon noticing a prized plant has wilted, doesn’t panic, but simply fetches the watering can.
Here’s the skeleton of it, written in a language any system would understand:
#!/bin/bash
SERVICE="my-precious-service"
if ! pgrep -f "$SERVICE" > /dev/null; then
echo "$(date): $SERVICE is down. Attempting restart." >> /var/log/watchdog.log
systemctl start "$SERVICE" || {
echo "$(date): Failed to restart $SERVICE!" >> /var/log/watchdog.log
# Here, you might send an alert via a simple curl command to a monitoring service.
}
fi
The beauty of this script lies in its unassuming nature. It does one thing, and it does it without fuss. You schedule it to run every minute via cron, and then you mostly forget about it. It becomes part of the background hum of the machine. The log file, /var/log/watchdog.log, remains empty for months, a testament to the service’s stability. Its only entries are brief, time-stamped notes of a job done: a service momentarily stumbled, was caught by the gardener, and set right again, often before any user could notice.
The deeper art, however, is in the restraint. A novice might be tempted to make the watchdog smarter—to check response times, validate database connections, or perform complex diagnostics. But each new line of logic is a new potential point of failure. The strength of the persistent gardener is in their singular focus. They don't diagnose the soil composition when a plant wilts; they water it. If the plant keeps wilting, the log fills up, signaling a deeper, more fundamental problem that demands your human attention.
This script is an act of humility. It acknowledges that our systems, no matter how carefully built, can and will fail in small, unpredictable ways. It’s not about building the un-crashable service; it’s about building a system that can pick itself up when it falls. It's a small, quiet promise to your future self and your users: even when you're not looking, something is.
Notes & further reading
A few pages I came back to while writing this: