The Conscientious Compactor: A Defence Against the Tyranny of the Trivial
This morning, my primary database server threw an alert. It wasn't a catastrophic failure, a security breach, or a runaway process. It was a simple, almost polite warning: disk usage at 85%. A quick investigation revealed the usual suspects. Not core application data, but the sprawling suburbs of operational logs. They had grown, quietly and predictably, into a bloated archive of everything from successful user logins to millions of debug entries from a long-forgotten feature flag. The useful signal was buried under an avalanche of noise.
We often treat logging as a binary decision: on or off. We set a log level to DEBUG or INFO, point our logs to a file, and forget about them until, like a neglected garage, they become so full we can’t find anything useful. This is the tyranny of the trivial—the assumption that more data is always better. But a log entry without a purpose is not just wasted space; it's a liability. It slows down searches, increases backup costs, and creates a compliance risk if it accidentally records something it shouldn't. The true discipline of logging isn’t just in the collection, but in the curation.
The Log as a Living Document, Not a Dumping Ground
The technique I want to propose is simple, manual, and profoundly effective. I call it the "conscientious compaction." Once a quarter, I perform a ritual that has become as crucial as any backup test. I pull up the log configuration for every service I manage, and I ask a single, ruthless question of every log statement: "What would I actually do if I saw this message?"
This isn't about turning off logging. It's about elevating its intent. That DEBUG entry that dumps a full user object on every API call? It was useful for two days three years ago. It's now just pumping kilobytes of personal data into my archives for no reason. It gets relegated to TRACE level, or removed entirely. That INFO message that cheerfully announces "Cache refreshed successfully" every five minutes? If the cache refresh failing is critical, then the log should be an ERROR when it fails. A continuous stream of success messages is just background static. I change it to log only on failure, or reduce its frequency to a daily heartbeat.
The goal is not silence, but clarity. After this compaction, what remains are logs that demand action (ERRORs), logs that help stitch together a story during an incident (WARNings and contextual INFO), and a very narrow, purposeful set of DEBUG logs for active development or troubleshooting. The result is a log file that you can actually read, where a new line appearing is an event worthy of attention, not just another drop in a torrent.
This practice forces a valuable conversation with your own systems. It’s a form of ongoing documentation, revealing which parts of your application are truly understood and which are just running on autopilot. The disk space you reclaim is a nice bonus, but the real victory is the regained focus. You are no longer a passive collector of data, but an active curator of meaning. Your logs become a sharp, reliable tool, a testament to the quiet virtue of keeping only what truly matters.
Notes & further reading
A few pages I came back to while writing this: