The Unwatched Pot: Why Less Monitoring Might Be More Reliable
We are taught, from our first days tinkering with a server, that vigilance is the highest virtue. The common wisdom is unassailable: instrument everything, alert on every deviation, and watch the dashboards with the unwavering focus of a lighthouse keeper. We build sprawling panopticons of Prometheus, Grafana, and PagerDuty, convinced that this intricate web of observation is the very fabric of reliability. But what if this obsessive monitoring is, in some quiet way, making our systems less reliable, not more?
Consider the human cost. Every alert, no matter how trivial, carries a cognitive tax. A flood of low-priority notifications—a brief latency spike, a single failed health check, a disk reaching 80% capacity—creates a background hum of anxiety. This noise doesn't just annoy; it actively trains us to ignore the console. It fosters alert fatigue, a condition where the truly critical alert, the one that screams "fire," is lost in the cacophony of whispers crying "maybe a slight draft." Our finely tuned observation system, designed for maximum awareness, ironically breeds a state of deafness.
This leads to a more insidious problem: the fragility of the observer. A system that requires constant human interpretation and intervention for every minor event is a brittle system. We become the shaky, error-prone component in the loop. We scramble to diagnose a blip that resolves itself moments later, often causing more disruption through our frantic actions than the original event ever could. We are like gardeners who pull up every slightly discolored leaf, damaging the plant in our quest for perfect health.
The Argument for Strategic Neglect
The counterintuitive alternative is to embrace a form of strategic neglect. This isn't about being blind; it's about being selectively sighted. It means designing systems that are truly autonomous for a wider range of failure states. Instead of alerting on a process failure, build a system that automatically restarts it. Instead of paging someone for a full disk, script the log rotation and cleanup, and only alert if the automated remedy fails.
Focus monitoring ruthlessly on symptoms that truly matter to the end-user, not on the internal mechanics that might lead there. Is the service returning 200 OK? Is the response time within SLO? These are the signals that matter. The rest is often just noise. By building more self-healing capacity and reducing the number of alerts to only those that require a conscious, human decision, we create a calmer, more sustainable operations environment. The pot, left unwatched, boils just fine. And when it doesn’t, the sound it makes is so distinct and urgent that we cannot help but hear it.
Notes & further reading
A few pages I came back to while writing this: