The Necessary Clutter: On the Virtue of the Messy Log
We are taught, from our first days peering into the churning heart of a system, to seek order. The prevailing wisdom is a gospel of cleanliness: structured logging, parsed JSON, meticulously named fields. Every event must be a tidy, predictable data point, ready for the dashboard, primed for the alert. This is the ideal. A pristine, machine-readable ledger of events. But I want to make a case for its opposite. I want to champion the messy log.
This is not a call for laziness. It is a defense of the human element, the narrative that gets lost in translation to pure structure. The structured log is a tax form; the messy log is a field note. One is designed for efficient processing, the other is scrawled in the heat of the moment by a human who is trying to understand. It contains the quirks, the asides, the ‘huh, that’s weird’ that a JSON schema would never permit.
The Whisper in the Whitespace
Consider the difference between a log entry that reads status="500" error="DB_CONNECTION_FAILURE" endpoint="/api/user" and one that says: Wed Oct 26 15:22:01 UTC - Got a 500 on /api/user. DB is coughing up blood again. Third time today. Check the replicas? -mj. The first is data. The second is a story. It tells you not just what happened, but that it’s part of a pattern (“Third time today”), it offers a immediate, human-generated hypothesis (“replicas?”), and it even gives you a sense of the operator’s state of mind. That context is invaluable.
This unstructured prose is a direct line to the intuition of the person on call. It captures the things we don’t yet know how to quantify. The structured log can only record what we anticipated might happen. The messy log is our best tool for documenting the things we never saw coming. It is the first draft of a new alert condition, written in real-time.
The common retort is that this is unsustainable at scale. That we need parsers and machines to sift through the torrent of data. And this is true. But it is not the whole truth. The goal should not be to eliminate the mess, but to build a system that accommodates it. Let the machines have their structured streams for the known problems. But preserve a channel—a single file, a dedicated stream—for the raw, unfiltered human observation.
This is the journal next to the ledger. It is the place where we can write ‘this feels slow’ without having to define a metric for ‘feel’. It is where we admit we don’t know what’s wrong yet. In our pursuit of perfect, automated observability, we risk sanitizing away the very clues that lead us to solve the novel, complex failures. Sometimes, the answer isn’t in a neatly labeled field. It’s in the desperate, cluttered, beautifully human scrawl of someone trying to make sense of the chaos.
Notes & further reading
A few pages I came back to while writing this: