The Unwitnessed Fall: On the Log Entry That Was Never Written
We spend our days crafting systems to tell us stories. We configure loggers to whisper every secret, every tremor, every sigh of our services. We build elaborate dashboards to translate these whispers into a narrative we can understand at a glance. A spike here is a surge of traffic; a flatline there is a silent alarm. We become connoisseurs of these tales, learning to read the water of a running system like a riverboat pilot. But what of the stories that are never told? What of the event that happens in the forest of our infrastructure when no logger is around to hear it?
I am not talking about a logging failure, where the pipe is clogged and events are lost in a buffer somewhere. That is a different tragedy, one of neglect. No, I am thinking of the event that is, by its very nature, unwitnessable. The single, silent failure that leaves no trace precisely because it was the thing that was supposed to do the tracing.
Consider the humble watchdog. Its entire existence is a promise: if the main service falters, it will log the failure and restart it. We see the restart in the logs and we nod, satisfied the system is working as designed. But what if the watchdog itself is the one that trips? It doesn’t crash dramatically; it just… stops. Its scheduled cron job misses a beat. Its process is terminated by a rogue out-of-memory killer. It enters a state of profound, silent confusion from which it cannot recover.
There is no error message for its own absence. The log file for that hour is simply blank, not out of error, but out of nothingness. The service it was guarding may still be running, for now, a ghost ship sailing on with a dead crew. Or it may have failed moments after its keeper vanished, its own demise now equally unrecorded. The system appears, from the outside, to be in a perfect, steady state. The graphs are flat. The alerts are silent. It is the most dangerous kind of failure: one that presents itself as perfect stability.
This is the unwitnessed fall. It teaches a humbling lesson about the layers of our observability. We must build systems that not only report on the state of the world, but also on the state of their own ability to report. It’s the difference between a sentry who shouts when he sees the enemy, and a sentry who must also shout at regular intervals to confirm he is still awake and at his post. Our watchdogs need watchdogs, and theirs need watchdogs too, a recursive descent into ensuring that someone is always there to witness the event, even if the event is the failure of the witness.
It is a pursuit that can feel philosophical, even absurd. But in the quiet, unlogged moments where our systems truly fail, this is the boring, reliable work that separates a system that is merely running from one that is truly watched.
Notes & further reading
A few pages I came back to while writing this: