The Janitor and the Scout: A Tale of Two Log-Monitoring Strategies
There are two archetypes that emerge when you’re responsible for keeping a small service running. You might find yourself, as I have at different times, embodying both, sometimes in the same afternoon. The first is the Janitor. The second is the Scout. They represent fundamentally different approaches to the humble, critical task of monitoring logs, and the choice between them shapes not just your tools, but your entire relationship with the systems you tend.
The Janitor is reactive and methodical. Their primary tool is the alert—a specific, tripped wire that signals a known problem. A disk usage threshold at 90%, a process that has stopped responding to heartbeats, a sudden spike in 5xx errors. The Janitor’s philosophy is one of order and known quantities. They have a mental, or ideally an automated, checklist for each alarm. Their world is a series of contained messes to be cleaned up. It’s a defensive stance, honed by the experience of past failures. There’s immense value in this; a good Janitor brings a powerful predictability to operations. The system is a known territory, and their job is to keep the paths clear of the usual debris.
In stark contrast, the Scout is exploratory and proactive. Their tool is not the alarm bell, but the pattern, the anomaly, the gentle drift. The Scout spends less time setting rigid thresholds and more time simply reading the logs, not to solve a problem, but to understand the landscape. They notice that a particular background task is taking a few milliseconds longer to complete each week, a slow accretion of slowness that would never trigger a critical alert. They spot an unfamiliar, low-frequency error code that appears once a day, a ghost in the machine that the Janitor’s alarms, tuned to volume, would ignore. The Scout’s goal is not just to fix what’s broken, but to learn what ‘broken’ might look like before it arrives.
The risk for the Janitor is that they become a prisoner of their own alarms. They only know what they’ve been told to look for. A novel failure mode, one that doesn't fit the predefined patterns, can slip by entirely until it becomes a crisis. The system becomes a black box that only speaks when something is grievously wrong, and the resulting relationship is one of stressful reaction. The Scout, meanwhile, risks disappearing down rabbit holes. Not every anomaly is a portent of doom; some are just statistical noise. A Scout without discipline can spend days chasing phantoms, neglecting the obvious, tangible maintenance the Janitor handles with such ease.
The most effective stewardship, I’ve found, lies not in choosing one over the other, but in building a rhythm between the two. Use the Janitor’s methods to create a solid, reliable baseline—a system that automatically handles the predictable failures. This creates the necessary calm, the stable foundation, from which the Scout can operate. Once the known problems are on autopilot, you free up the mental bandwidth to be a Scout: to browse the logs with curiosity, to ask “what’s different today?” without the pressure of a siren blaring in your ear. It is the Scout’s discoveries that then inform the Janitor’s future checklists, turning novel anomalies into known, manageable events. The Scout finds the new path, and the Janitor paves it.
Notes & further reading
A few pages I came back to while writing this: