Against Alerts in All-Caps
There’s a piece of received wisdom in operations so deeply ingrained we barely think to question it: the alert that demands your attention must be *shouted*. It must be red, it must be bold, it must be in all-caps. CRITICAL. SEV-1. SYSTEM DOWN. We build dashboards that look like air traffic control for a thunderstorm, convinced that if the visual signal isn’t aggressive enough, we will become complacent, we will miss the one signal in the noise.
But I want to challenge this. I want to suggest that the all-caps alert, the screaming siren in our console, is not just an aesthetic choice—it’s a philosophical one that erodes our capacity for calm response. It trains us for panic, not for thought. When every ‘warning’ is a shout and every ‘critical’ is a fire alarm, we begin to treat them all with the same blunt instrument: a frenzied scramble to make the noise stop. The subtle, important distinctions between a failing cache and a failing database are lost in the uniform volume of the alarm.
The Boy Who Cried Wolf, in Helvetica Bold
This practice creates a kind of alert inflation. If a minor service degradation is flagged as ‘MAJOR OUTAGE,’ what language is left for the true, cascading failure? We exhaust our own emotional and cognitive vocabulary. The team becomes desensitized, a condition sometimes misdiagnosed as ‘alert fatigue,’ but which is really ‘context fatigue.’ The signal hasn’t been drowned out by volume; it’s been made meaningless by a lack of nuance.
Consider the alternative: a quiet, persistent signal. A service designed to fail gracefully might change its status icon to a steady amber, not a flashing red. A log line might read, with unsettling calmness, “Primary database connection unavailable, failover initiated.” This is information, not an emotional trigger. It trusts the operator to be competent, to read, to assess. It says, “Here is the state of the system,” not “YOU MUST FEEL AFRAID NOW.”
The tools we use shape the culture of the teams that wield them. A team conditioned by shouted alerts develops a firefighting mentality, valuing speed over understanding, reaction over diagnosis. A team served by clear, calm signals cultivates a culture of analysis. They are detectives, not first responders. They are more likely to ask “why?” before they ask “how fast can we patch it?”
This isn’t an argument for removing urgency, but for redefining it. True urgency should be communicated through clarity of impact and scope, not typographic violence. Let the ‘critical’ alert be rare, and let its gravity be conveyed by the concise, precise description of what is broken and for whom—not by the size of its font. The most reliable system I ever worked on had a single, stark rule for the ops channel: if you get a message here, it means a user transaction is failing right now. The messages were plain text. We never missed one. We never had to shout.
Perhaps reliability isn’t just about the systems we watch, but about the state of mind we maintain while watching them. Ditch the caps lock. Speak clearly, and trust your crew to listen.
Notes & further reading
A few pages I came back to while writing this:
- Chattanooga, TN
- The Ghost File: A Simple Trick for Testing Service Lifecycle
- Memphis, TN
- The Unwatched Pot: Why Less Monitoring Might Be More Reliable
- Nashville, TN
- The Mariner's Logbook: Obsolescence in the Data Stream
- Amarillo, TX
- Austin, TX
- Brownsville, TX
- Carrollton, TX
- Corpus Christi, TX
- Dallas, TX
- Fort Worth, TX