The Deliberate Scribe: How a Single Script Can Unify a Messy Logscape
Every system, in time, grows its own unique forest of logs. A cron job scatters messages to syslog. A bespoke service writes to a flat file in /opt. A database daemon buries its history in a binary format. A web server rotates its access logs daily. Individually, they make sense. Collectively, they form a fragmented chronicle of your infrastructure, a story written by a dozen different authors in a dozen different dialects. Attempting to correlate a user’s journey through this thicket during a moment of crisis is a test of both patience and sanity.
We often dream of monolithic, all-seeing observability platforms to solve this, but today I want to talk about something simpler, more immediate, and often more resilient: the art of deliberate, unified scribing. The technique is this: a single, short shell script, run as a cron job every minute, whose sole purpose is to gather state from across the system and write a single, consistent line to one log file.
This isn’t about replacing your existing logs. It’s about adding a common thread. My script, simply named /usr/local/bin/scribe, does nothing more than assemble a brief report. It uses curl to hit the local health endpoint of a service and record the HTTP code. It parses the output of pg_isready to check the database connection. It counts the number of entries in a processing queue directory. It checks the load average. Each of these checks is a simple command substitution, its output captured into a variable.
The magic is in the final, single echo statement. It writes a line to /var/log/scribe/status.log with a strict format: a timestamp, the hostname, and then a pipe-delimited string of key-value pairs. The line might look something like this: 2024-05-21T10:01:03Z host-web-01 http=200|db=accepting|queue_size=14|load=0.12. That’s it. One line, per minute, per server.
The Compounding Power of Consistency
The immediate benefit is obvious. When something goes wrong, you have a single, high-level source of truth. You can tail -f this one file and watch the health of the system pulse by. But the subtle, long-term power is in the consistency. This log file is predictable. It has a fixed, simple schema. Over months and years, as services are rewritten in new languages and new components are added, the scribe script is updated to include them, but the log format remains. It becomes the Rosetta Stone for your system's history.
This approach also forces a healthy discipline. Every time you add a new service, you are compelled to ask: "What is the one command that tells me if this is working?" You are defining a concrete health check, a contract with the machine. The script itself becomes a living document of what you, the operator, deem important enough to monitor every sixty seconds.
Is this a replacement for detailed, high-resolution metrics? Of course not. But it is an anchor. In the storm of a complex outage, when your fancy dashboards are lagging or themselves failing, you can SSH to a box, type a single command, and get a human-readable, time-ordered summary of the last hour. It’s the operational equivalent of a ship’s logbook: handwritten, deliberate, and unfailingly reliable. It's the deliberate act of writing one simple line that tames the chaos of the noisy forest.
Notes & further reading
A few pages I came back to while writing this: