The Ancient Bell-Ringer and the New Sirens: On Alerting vs. Noticing
There is a quiet, fundamental schism in the way we watch over our systems, a divide that runs deeper than any choice of software or protocol. On one side stands the Alert, a precise, automated siren designed to startle us into action. On the other stands the Log, a patient chronicle from which we must learn to notice the subtle patterns of decay. One demands an immediate response; the other requires a slow, deliberate conversation with the machine. The tools we choose, and the philosophy we embed in our operations, often place us in the camp of either the bell-ringer or the cartographer, and the distinction shapes everything.
The modern sirens—our PagerDutys and OpsGenies—are marvels of engineering efficiency. They are built around the concept of the threshold: when this metric crosses that line, sound the alarm. This approach is the descendant of the factory whistle, signalling a broken loom. It creates a clear contract: no news is good news. For a service that must maintain five-nines uptime, this is non-negotiable. A latency spike or a crash-looping service demands immediate, sometimes frantic, intervention. The siren’s great strength is its clarity of purpose. Its great weakness is its inherent noise. It can only cry wolf so many times before the operators, the villagers, begin to ignore its call, or worse, become numb to the constant state of emergency it creates.
Contrast this with the ancient practice of the bell-ringer, not as an alarmist, but as a timekeeper. The bell rings not because something is wrong, but because it is time to pay attention. This is the realm of the log aggregator, the dashboard, the scheduled ‘cat’ of a critical file. There is no piercing siren, only the steady, predictable tolling that invites a glance, a moment of reflection. The operator who spends thirty minutes each morning scrolling through logs is not fighting fires; they are walking the perimeter, looking for footprints in the dew. They are noticing that the database connection pool is draining a little faster than it did last week, or that a particular user agent is starting to make curiously structured requests. This is not alerting; it is noticing. It is the cultivation of a deep, intimate familiarity with the system’s normal breath and pulse.
The sirens will always have their place. A server room on fire is not a subtlety to be noticed during a morning review. But an over-reliance on alerts creates a fragile system governed by reactive panic. It outsources awareness to a set of brittle rules. The practice of noticing, however, builds a resilient operator and, by extension, a more resilient system. It fosters a proactive intuition, an ability to sense the tectonic shifts before they become earthquakes. The goal is not to eliminate the sirens, but to quiet them enough that when they do scream, the silence they break is profound and meaningful. The ultimate state of ops is not a silent console free of alarms, but a quiet hum punctuated only by the necessary, respected bell, and the informed gaze of someone who knows what to look for before the threshold is ever crossed.
Notes & further reading
A few pages I came back to while writing this: