The Deliberate Blind Spot: When to Stop Logging
We are told, as a near-religious commandment, to log everything. Every request, every state change, every error, every heartbeat. The logic is seductive: with enough data, any post-mortem becomes a simple forensic exercise. We build our observability stacks like vast cathedrals to comprehension, believing that if we can just capture enough light, no shadow can hide a bug. But I want to propose a heretical thought: that the most reliable systems often contain intentional, reasoned gaps in their vision. That sometimes, the path to stability is to deliberately choose not to know.
Consider the humble home thermostat. It does not log the temperature of every room in your house for the last six months. It does not alert you when the differential from the setpoint drifts by 0.5 degrees. It knows one thing: right now, is it too cold, or too hot? It acts on that, and then it rests. Its simplicity is its reliability. We, in our digital domains, have forgotten this economy of attention. We instrument our services to such a degree that the monitoring itself becomes a source of load, complexity, and—ironically—failure. The logs about the logging system become a significant portion of the logs.
This isn't an argument for negligence. It's an argument for design. Every log line has a cost: storage, ingestion, indexing, and, most critically, human attention. When you log everything, you effectively log nothing, because the signal drowns in a self-imposed noise. The ‘alert fatigue’ we lament is often a direct result of our own compulsion to document every twitch and tremor. A system that cries wolf a hundred times a day is eventually ignored, and that is when the real wolf gets through.
Cultivating a Bias for Action, Not Archival
The counter-intuitive practice is this: for each component, define not what to log, but what you will do with the log. If there is no clear, pre-defined action—no automated response, no human procedure, no architectural decision it informs—then question its existence. Is the `DEBUG` level chatter from a library you don't own truly making your service more reliable? Or is it just digital hoarding, a ‘just-in-case’ pile that makes finding the actual needle exponentially harder?
Embrace the deliberate blind spot. A cron job that runs daily and succeeds 99.9% of the time might not need its success logged at all; its failure is the only event that matters. A high-volume, idempotent API endpoint might only need metrics for latency and error rates, not a forensic trail of every single transaction. The goal is to build systems where the logs you do keep are so potent, so full of signal, that reading them is a clarity, not an archaeological dig.
Reliability emerges from clear, robust feedback loops, not from exhaustive surveillance. By strategically choosing where to look away, we focus our gaze—and our response—on what truly matters. We build systems that, like the thermostat, are designed to act and be understood, not to merely record their existence for a posterity that will never come. The quietest, most boring machine is often the one that knows precisely what to ignore.
Notes & further reading
A few pages I came back to while writing this: