The Two Gardeners: On the Deliberate Pruning and the Wild Growth of Logging
There are, I’ve come to believe, two distinct philosophies that govern our approach to logging in a small service. They are not merely technical choices, but aesthetic ones, born from a deep-seated belief about the nature of the systems we tend. I think of them as the two gardeners.
The first is the Gardener of the Formal Garden. This practitioner believes in order, clarity, and intent. For them, every log line is a deliberate planting. They define schemas with care, structuring JSON logs with precise key-value pairs: user_id, event_type, duration_ms. Their logs are a pristine hedge, trimmed to perfection. The primary tool is aggregation; they know exactly what they are looking for because they decided what to grow in the first place. A query for the 95th percentile response time for a specific endpoint is a simple, direct path through the garden. The beauty here is in the predictability and the ease of answering the questions you knew to ask.
The second is the Gardener of the Wild Meadow. This one sows seeds broadly and trusts in the emergent patterns of nature. Their philosophy is one of abundance and discovery. They log liberally—debug statements, stack traces on warnings, entire request/response payloads. Their logs are a dense, thriving meadow of text, rich with context and the unexpected. The primary tool here is not aggregation, but grep; it is the act of wandering through the tall grass, looking for a specific flower or an unusual insect. The value is not in answering a predefined question, but in stumbling upon the question you didn’t know needed asking.
Each approach has its perils. The Formal Garden, for all its elegance, can be brittle. When a novel fault occurs—a failure mode you never conceived of—your pristine schema may offer no context, no clue. You are left staring at a perfectly trimmed hedge, wondering what creature just scurried through a hole you didn't know was there. The Wild Meadow, meanwhile, risks becoming an impenetrable thicket. The sheer volume can obscure signal with noise, and the cost—both in storage and in cognitive load to parse it—can grow untamed.
The wisdom, as with most things, lies not in choosing one absolutely, but in understanding the season. The Formal Gardener builds for the long summer of normal operation, where efficiency and monitoring reign. The Wild Gardener is essential in the sudden storm of a novel outage, where the clues are hidden in the undergrowth. A mature service, like a mature landscape, needs both: the deliberate, predictable paths of the formal garden for daily tending, and the wild, untamed meadow at its edges, preserving the raw, unedited truth for when we must go searching for answers we do not yet possess.
Notes & further reading
A few pages I came back to while writing this: