The Quiet Custodian vs. The Noisy Tenant: On Two Kinds of Backup Alerts
There is a moment, late every evening, when a small green light on a machine in the corner of my office blinks twice and goes dark. It’s the signal. A silent, successful handshake between a server and its off-site sibling. The day’s work is tucked away. No fanfare, no email, no push notification. It is the work of a quiet custodian, doing its job with a solemn, understated grace.
This stands in stark contrast to another system I inherited, one that announces its success with the fervor of a noisy tenant clanging pipes in celebration. Every completed task, every verified checksum, triggers a cascade of alerts into a Slack channel. “BACKUP SUCCESSFUL,” it declares, over and over, a digital parade for a ceremony that happens 365 times a year. The messages, once urgent signals, have become background noise, scrolled past without a glance.
These two approaches represent a fundamental philosophical divide in how we design our systems to communicate their health. The Quiet Custodian operates on a principle of silent integrity. Its design assumes that no news is good news. It logs its actions meticulously for the day we need to audit its trail, but it only cries out when it fails to perform its duty. This approach demands trust and a different kind of vigilance from us. We must proactively check its logs periodically, not because we doubt it, but as a ritual of respect for its quiet work. The relationship is built on a steady, quiet confidence.
The Noisy Tenant, however, is built on a principle of verification through announcement. It seeks to prove its worth constantly, shouting its accomplishments into the void lest anyone forget it is there, working. This creates a brittle kind of trust—we believe it works only because we are constantly told it works. The danger, of course, is alert fatigue. The crucial alert of a genuine failure is lost in the cacophony of its successful brethren, all dressed in the same urgent clothing. We stop listening to the tenant who cries "wolf" for every passing sheep.
I find myself drawn to the custodian’s way. There is a profound humility in a system that does its essential work without demanding applause. It treats reliability not as an exception to be celebrated, but as a baseline to be expected. Our attention, a scarce resource, is reserved for the moments of true exception—the missed handshake, the failed checksum, the silent green light that never blinks. In that silence, we find a clearer signal, and in that reserved alert, a far more powerful and meaningful alarm.
Notes & further reading
A few pages I came back to while writing this: