The Unseen Courier: On the Quiet Duty of the Log Shipper's Heartbeat
In the quiet hum of our small services, the log shipper works its silent trade. It is the unseen courier, the faithful runner of messages from the outposts of our systems to the central hall where we listen. We configure it, start it, and often forget it, trusting it to perform its duty without fanfare. But how do we know it is still there, still running, still carrying our most crucial whispers? A process that has crashed cannot log its own failure.
The most elegant solution I have found is not a complex monitoring rule or a heavy-weight agent, but a simple, self-perpetuating echo. A heartbeat. The technique is this: configure your log shipper—be it Filebeat, Vector, Fluentd, or another—to also watch a small, local file. Then, from the same machine, use a cron job to regularly append a single, distinct line to that very file.
This creates a closed loop, a tiny Ouroboros of assurance. The cron job, that most ancient and reliable of sentinels, writes its message. The log shipper, ever-vigilant, picks it up and dispatches it to its destination—your logging dashboard, your alerting system, your archival storage. The presence of this regular pulse in your log stream is the proof of life. Its absence is the alarm.
Crafting the Pulse
The implementation is beautifully simple. First, add a new entry to your shipper's configuration, pointing it at a file like /var/log/heartbeat.log. Then, craft a cron entry: * * * * * echo "HEARTBEAT $(hostname) $(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> /var/log/heartbeat.log. This will write a timestamped message every single minute.
Now, in your central logging system, you can set an alert that triggers if the heartbeat from any given server ceases for more than, say, two minutes. The failure could be the shipper process itself, the cron daemon, a full disk, or a network partition. The specific cause matters less in the initial moment than the simple, stark truth the heartbeat reveals: the messages from this server have stopped. The courier has fallen silent.
This method borrows stability from the very components it watches over. It uses the constancy of cron to verify the vigilance of the shipper. It is a testament to the idea that the most reliable systems are often built from simple, interdependent checks and balances. They are not monolithic giants but communities of small, trusting services, each one whispering its continued existence to the others, ensuring that no failure goes entirely unseen, and no courier's silence goes unheard.
Notes & further reading
A few pages I came back to while writing this: