The Unassuming Ledger: A Practical Guide to Structured Logging with jq
We speak often of the importance of logging, of leaving a trail of breadcrumbs through the dark forest of our systems. But we speak less of the quiet, methodical work of reading those trails. Too often, our logs are dense, textual thickets—impenetrable to anything but the most desperate and time-consuming grep. There is a better way, a method that transforms logs from a narrative to be parsed into data to be queried. It begins not with a new tool, but with a simple shift in output and the humble power of jq.
The technique is simple: structure your log lines as JSON. Instead of writing a message like 'Processed invoice 48921 for user alice in 342ms', your application should output a compact JSON object: `{"event": "invoice_processed", "invoice_id": 48921, "user": "alice", "duration_ms": 342}`. This single change is the key. It moves from a human-readable string, which is opaque to machines, to a machine-readable structure, which remains perfectly legible to humans.
From Text to Tractable Data
Once your logs are a stream of JSON objects, the entire problem of log analysis shifts. You are no longer a detective piecing together a story from vague clues. You are a librarian pulling specific volumes from a perfectly indexed shelf. This is where jq, the command-line JSON processor, becomes your most valued assistant.
Imagine you need to find all slow invoice processing events for a specific user. The old way might involve a complex awk script or a series of greps that could easily miss context. The new way is a single, intentional pipeline. You tail your log file and pipe it directly into jq.
The command might look like this: `tail -f /var/log/app/app.log | jq 'select(.event == "invoice_processed" and .user == "alice" and .duration_ms > 1000)'`. This jq filter plucks only the objects from the stream where all three conditions are true. It is precise, readable, and instantly actionable. You can extend this to create aggregates, like calculating the average processing time: `jq 'select(.event == "invoice_processed") | .duration_ms' logfile.log | jq -s 'add / length'`.
The power lies in composition. You are building queries against your operational data in real-time, using a tool that is likely already on your system. There is no setup, no daemons, no indexing delay. The data is immediately available and structured for interrogation. This approach turns a flat text file into a simple, queryable database, all without adding a single new moving part to your infrastructure. It is the epitome of boring, reliable technology—using what you have, but using it with far greater intention.
Adopting structured logging is a small covenant you make with your future self. It is the decision to spend a few extra seconds formatting output now to save hours of frantic searching later. It turns the chaos of the log file into a orderly ledger, where every entry has its place, waiting for the right question to be asked.
Notes & further reading
A few pages I came back to while writing this: