The Unseen Cartographer: On Mapping the Known Unknowns of Your Own System
A reader wrote in recently with a question that felt both simple and profound: "I can see everything my small service is doing. The logs are verbose, the metrics are collected. But how do I know what I'm supposed to be looking for? How do I know what normal even looks like?"
This is the cartographer's dilemma. You have the tools to draw a detailed map of every hill and valley, every river and road of your system's behaviour. But a map is useless without a sense of scale, without a north star, without the faint, dotted lines that mark the well-trodden paths. You are staring at a landscape of pure data, but you lack the legend to interpret it. You are surrounded by known facts, but you are navigating the territory of the 'known unknown'—you know what you don't know.
The answer, I've found, isn't in more logging or more dashboards. It's in a quieter, more deliberate practice: the creation of a baseline. Before a crisis, during a period of quiet reliability, is when you must become the cartographer of your own system's normalcy. This isn't about setting rigid performance thresholds or error budgets. It's about observation. It's about spending a week, or even just a day, simply watching the system work.
Watch the daily rhythm. Note the gentle hum of cron jobs at 3 AM. See the small, predictable spike in traffic every weekday at 9:05. Observe the memory usage graph as it climbs slowly through the day and is gently reset by a nightly process. Document the specific, harmless warning that always appears on a cold start. This is the topography of your system's peace.
You are sketching the coastline of the known world. The value of this map isn't realized during calm seas; its power is revealed only when the weather turns. When an alert fires at 2 PM on a Tuesday, you won't just see a spike—you'll see an anomaly that breaks the rhythm you've committed to memory. That harmless warning will be joined by a new, sinister cousin, and you will notice its unfamiliar face immediately. The 'known unknown' becomes a 'known'. You are no longer just collecting data; you are recognising a deviation from a pattern you yourself defined.
This practice transforms you from a passive collector of logs into an active interpreter of your system's story. The logs become a narrative you can read, and the metrics become a language you can speak. You stop asking "Is something wrong?" and start asking the far more powerful question: "What is different?" And that is the first, and most crucial, step toward an answer.
Notes & further reading
A few pages I came back to while writing this: