The Glorious Chaos of the Unfiled Log
We preach filing. We venerate the clean, parsed, indexed, and searchable log stream. We invest in tools that slice our system’s mutterings into tables, dashboards, and alert rules. The goal is total legibility: to transform the raw, messy chorus of a running machine into a sterile document. We call this observability. I’ve come to think of it, in its most rigid form, as a kind of deafness.
My counterintuitive proposal is this: you need at least one crucial log file that you never, ever parse. A single stream of pure, unadulterated text, written sequentially, appended to in perpetuity, and read only by human eyes scrolling through it, start to finish, when something is deeply wrong. This is your Unfiled Log. It is not for machines. It is for you, the operator, to swim in the context of the system as it actually is, not as your schemas dictate it should be.
The Pattern You Weren't Looking For
Parsed logs are brilliant for finding what you’re looking for. They are terrible for finding what you aren’t looking for. The correlation engine will show you that the API latency spike coincided with the cache miss. It will not show you the faint, weird punctuation of a rare external DNS timeout that appears three lines above every third database connection warning—a pattern only visible in the raw, unsegmented flow of time. Our tools, by design, filter out the “noise.” But in a complex system, the signal is often buried in what we’ve prematurely dismissed as noise.
Reading the Unfiled Log is a tactile, slow practice. It’s the difference between reading a summary of a novel and reading the novel itself. You feel the rhythm of the system. You see the casual interleaving of cron jobs, health checks, and user requests. You notice the orphaned debug statement someone left in six months ago. You develop a sense of the system’s “normal” texture, so the introduction of a new, subtle abnormality—a slight change in timing, a new type of message you didn’t instrument for—feels immediately off, like a single wrong note in a familiar song.
This practice defends against the arrogance of presumed knowledge. When we define fields and tags, we are making a bet: we are betting that we already understand what is important. The Unfiled Log is a hedge against that bet. It is a humble admission that our models of the system are incomplete, and that true understanding sometimes requires wading into the glorious, untamed chaos of reality itself. It is the ops equivalent of walking the factory floor, instead of just reading the production reports.
So, keep your Elastic cluster. Cherish your metrics pipeline. But somewhere, on a durable disk, let one honest-to-goodness, timestamped, plain-text log file run like a river. Don’t index it. Don’t structure it. Just let it be. And when the finely-tuned alerts fail to tell you why, sit down with a cup of coffee, open a terminal, and read. You might be surprised at what the system has been trying to tell you all along, in its own, unfiltered voice.
Notes & further reading
A few pages I came back to while writing this: