The Unnecessary Elegance of the Redundant Loop
We are taught, from our first days tending to systems, that redundancy is a virtue. We build it into our storage arrays, our network paths, our power supplies. The mantra is simple: two is one, and one is none. It is a doctrine of survival, and it is not wrong. But there is a quieter, more insidious form of redundancy we practice without question, one that offers the illusion of safety while quietly eroding our understanding: the redundant notification loop.
Consider the standard alerting pipeline. A service fails. A monitoring tool detects the failure and fires an alert to a central dashboard. That dashboard, in turn, triggers a push notification to a phone, an email to a distribution list, and a message into a dedicated chat channel. We have created a perfect, multi-layered echo chamber. The same human, the same on-call engineer, receives the same signal through three different mediums simultaneously. Their phone buzzes, their laptop chimes, and their watch taps their wrist. It is a symphony of urgency, but the melody is just one note, played over and over.
This is not resilience. This is noise. We have conflated the volume of the signal with its importance. The true purpose of an alert is to trigger a specific, intelligent human response. By bombarding a single point of human failure with identical data, we achieve the opposite. We induce alert fatigue, a state where the cacophony itself becomes the background hum, and the truly critical note is lost within it. The engineer begins to ignore the chat channel because the email is ‘more official,’ or dismisses the email because the push notification was first. The redundancy of the medium undermines the integrity of the message.
The Virtue of a Single, Clear Bell
The counterintuitive argument, then, is this: for a given event, there should be one authoritative notification path, and it should be respected as the sole source of truth. Choose one. Is it the chat ops channel? Then let that be the hallowed ground. Divert all other alerts to a silent, log-only state. Is it a paging system? Then let the page be the definitive call to arms, and suppress the chatter that would otherwise follow.
This requires a more thoughtful, less reflexive architecture. It demands that we classify events not just by severity, but by the required human response. A full service outage might merit the piercing ring of a page. A warning about rising memory usage might belong only in a dedicated monitoring channel, visible but not screaming. A completed backup? Perhaps it needs no notification at all, only a green checkmark in a log that someone might glance at on a Tuesday morning.
By dismantling the redundant loop, we force a clarity of purpose onto our systems. We are no longer building for the machine’s ability to shout, but for the human’s ability to listen and comprehend. The goal is not to ensure the alert is heard, but to ensure it is understood. And understanding, in the quiet panic of a failing system, is a far more valuable currency than noise.
Notes & further reading
A few pages I came back to while writing this: