The Quarry and the Quill: Recording Events vs. Interpreting Actions
There are two distinct rhythms to the data that flows from our services. One is the hard, percussive click of a turnstile, counting each passage with mechanical certainty. The other is the soft scratch of a pen, annotating the margins of a story with context and consequence. In the world of operations, this is the fundamental divide between two approaches to understanding our systems: the exhaustive recording of every event, and the curated interpretation of meaningful actions.
The first approach, which I think of as the Quarry, is about raw material. It's the philosophy of shipping every log line, every metric, every trace, every single state change into a vast data lake. The goal is completeness. The Quarry operates on the faith that if you have enough rock, you can always find the gem you need later. Tools that enable this are built for volume and ingestion speed; their primary virtue is fidelity. When a customer reports a bizarre bug that only occurred between 2:03 and 2:04 AM under a full moon, the Quarry assures you the data is there. You just have to mine for it.
The second approach, the Quill, is the opposite. It is sparing, deliberate, and narrative. The Quill doesn't record every heartbeat; it records the moment the pace quickened. It’s the difference between logging "DB connection pool request" a thousand times a second and writing a single, clear entry: "Initiating failover to secondary database cluster due to sustained latency threshold breach." The Quill demands upfront thought. It requires you to define what constitutes a significant action in your system's story, and to instrument *that*. Its virtue is clarity, but it risks being an incomplete chronicle, missing the subtle clues buried in the noise.
The tension between these two is palpable. The Quarry can feel like a wasteful obsession with detail, creating a tyranny of data where finding a signal requires sifting through mountains of noise. It's expensive to store and a nightmare to query when you don't know what you're looking for. The Quill, meanwhile, can feel arrogant. It presumes we are wise enough to know today what we will need to know tomorrow. When the unforeseen crisis strikes—the "black swan" event—the Quill may have left out the one, crucial, seemingly insignificant data point that explains everything, because it wasn't deemed important by the original scribe.
Most of us, of course, operate in a hybrid space. We let the Quarry run in the background, collecting low-level metrics and debug logs, while the Quill writes the high-level audit trail and business logic summaries. The real art lies in drawing the boundary between them. Which parts of your system are so critical that you need the Quarry's forensic-level detail? And where can you afford the elegant simplicity of the Quill? The answer isn't static. A new feature might start under the watchful eye of the Quarry until its behavior is understood, and then, as it matures, be entrusted to the more succinct annotations of the Quill. It’s a continuous negotiation between the fear of missing something and the burden of knowing too much.
Notes & further reading
A few pages I came back to while writing this: