The Two Weirs: On the Streams You Divert and the Flows You Let Run

There is a quiet, unglamorous fork in the road for anyone running a small service. It appears not during a crisis, but in the slow, methodical building of your observability. It’s the choice between what I’ve come to think of as the diverting weir and the measuring weir. Both manage the stream of events and logs flowing from your systems, but their philosophies, and their consequences, are worlds apart.

The Diverting Weir

The diverting weir is an act of preemptive curation. It’s the engineer, or the script, standing upstream with a sieve, deciding what is worthy of being recorded. It says: “Only errors above WARN go to the main channel. These particular user actions are business-critical; log them. These system pings are noise; discard them.” The logic is impeccable—reduce volume, lower storage costs, speed up queries by only keeping the “important” stuff. It’s efficient. It feels clean. You build a tidy, manicured log pond containing only the fish you intended to catch.

But the diverting weir has a silent cost. It operates on a map of the river drawn yesterday. When a new, subtle failure mode emerges—a slow degradation in a third-party API response, a strange pattern of semi-successful authentications from a new region—you may find its traces were sifted out long ago, deemed unworthy of the main channel. You are left diagnosing a problem with data you explicitly chose not to keep. The weir’s efficiency becomes a blindness.

The Measuring Weir

The measuring weir, by contrast, is an act of humble capture. It lets the entire stream flow into the reservoir, but it builds a simple gauge at the side. It says: “Let everything through, but count, sample, and index intelligently.” The verbose debug logs, the health pings, the trivial GET requests—they’re all there, stored cheaply in object storage or a low-cost tier, often in raw, barely-indexed form. The gauge—your primary monitoring—tracks the vital signs: error rates, latency percentiles, traffic volume.

This approach accepts the messiness of the real stream. Its power is not in daily tidiness, but in forensic possibility. When the gauge shows an anomaly, you can go to the full, undiverted record. You can ask new questions of old data. That bizarre error a user reported last Tuesday? The raw stream might still hold the context of the five requests that preceded it, context your diverting weir would have considered irrelevant noise and discarded. The measuring weir trades a bit of upfront storage complexity for a profound depth of hindsight.

The choice isn’t about right or wrong; it’s about temperament and scale. The diverting weir is the watchmaker, confident in the known interactions of the gears. The measuring weir is the gardener, knowing that while they tend the visible plants, the true cause of a blight might lie in the unseen soil. For a small service, the temptation is strong to over-divert, to keep things lean and understandable. But in doing so, we risk building a system that can only explain the failures we have already imagined. Sometimes, the most reliable technology is the one that has the humility to admit what it doesn’t yet know, and the patience to let the stream run, just in case.

Notes & further reading

A few pages I came back to while writing this: