The Unspoken Echo: A Simple Script to Prove Your Logs Are Listening
We talk a lot about shipping logs. We configure our agents, point them at our aggregators, and trust that the river of data flows uninterrupted. We build dashboards on top of this flow, confident that the light green 'OK' status means all is well. But how often do we test the assumption that the river is actually reaching the sea? We check that our backups are alive, but do we ever check that our logs are truly being heard?
There’s a quiet, unsettling gap between a service emitting a log line and that line being indexed and searchable in your central system. Network partitions, agent failures, misconfigured authentication, or quota errors can silently break the chain. You might only discover the break when you need to trace a problem and find a ghost town where yesterday’s bustling data should be. The silence isn't golden; it's blind.
The Canary in the Log Mine
The remedy is a simple, boring, and profoundly reliable technique: the log heartbeat. Once an hour, a cron job on a key server runs a script that does one thing and one thing only. It writes a uniquely identifiable message to the local system log. The message isn't an error or a warning; it’s a deliberate, purposeful echo. Something like: [HEARTBEAT][$HOSTNAME] Pulse confirmed at $(date -u).
This humble line gets picked up by your log shipper and sent on its journey to your central log archive. The magic happens in a second, independent process. Another cron job, perhaps on a different machine, runs a script that queries your log system—be it Elasticsearch, Graylog, or Splunk—searching for the most recent heartbeat from that specific server.
If it finds one within the last 65 minutes, it does nothing. Silence is success. If the search comes back empty, it triggers an alert. This isn't an alert about high CPU or low disk space; it’s a more fundamental alert. It says: "The narrative of this server has been lost. We are no longer listening to its story."
This creates a closed loop. It moves logging from a passive "set and forget" configuration to an active, verified system. You are no longer assuming your logs are working; you are proving it, continuously and automatically. It’s the equivalent of a watchman’s call in the night, and the echo that confirms they are still on duty. It turns the silent, assumed utility of logging into a spoken, verified certainty.
Notes & further reading
A few pages I came back to while writing this: