The Cartographer and the Compass: Plotting a Course Through Logs
There are two distinct philosophies for navigating the treacherous waters of a production incident. One seeks to draw a perfect, detailed map of the entire territory beforehand. The other trusts in a simple, reliable instrument to point the way when the fog rolls in. In the world of logging and operations, these philosophies manifest as the exhaustive, structured log event versus the humble, timeless log line.
The structured event is the cartographer’s masterpiece. It is a JSON object, rich with nested metadata, tracing IDs, and precisely typed fields. It dreams of being ingested by a powerful system that will automatically index, correlate, and visualize its contents. When it works, it’s a thing of beauty. You can query for all events where `user.tier` is ‘premium’ and `service.latency` exceeds 500ms, and a dashboard materializes the problem before you can finish your coffee. It is a map so complete you believe you can see every tree.
But maps can be outdated. They assume the landscape remains as it was surveyed. The structured log event is brittle. It depends entirely on the schema agreed upon by the application and the parsing engine. A change in a field name, a nested object that suddenly appears as a string, a new service that hasn’t been onboarded to the logging suite—these are rips in the parchment. During a crisis, when the very systems that parse these events are under duress, your beautiful map can dissolve into meaningless symbols.
This is where the compass comes in. The plain text log line is that compass. It is the unbreakable practice of writing a human-readable sentence, with a timestamp, that tells a simple story: ‘Could not connect to database XYZ on host ABC, retrying in 5s.’ It is not built for machines first; it is built for the weary human operator staring at a terminal at 3 AM. Its strength is its resilience. You don’t need a special parser, a schema, or even the original application to understand it. You can `grep` it, `tail` it, and `ssh` it from a jump box when everything else is down.
The choice is not absolute. The wise crew carries both map and compass. Critical, well-understood business events benefit immensely from structure, feeding our analytical tools for the calm days. But for the deep, operational health of a system—the kind that must be diagnosable when the world is on fire—we must never abandon the compass. It is the one tool that guarantees, even when all other systems have failed, that you can still tell which way is north.
Notes & further reading
A few pages I came back to while writing this: