The Two Shepherds: Tending Flocks with Fences and with Dogs
In the quiet pastures of our infrastructure, we are all shepherds. Our flock is the data, the services, the state of things. The perennial question is not whether to tend to them, but how. Two philosophies have emerged over the years, each with its own loyal following and its own distinct temperament. One builds a perfect fence. The other trains a very good dog.
The first shepherd is the architect of the fence. This approach is rooted in prevention, in designing systems that are, by their very nature, incapable of straying. It is the philosophy of immutable infrastructure, of read-only filesystems, of declarative configurations that are applied and then sealed. The fence is built once, tested thoroughly, and then trusted. Its goal is to create an environment where the flock simply cannot get into trouble because the possibility has been engineered out of existence. A service cannot overwrite its binary because the filesystem is locked. A configuration cannot drift because it is reapplied from a single source of truth with every deployment. The fence is static, silent, and absolute.
The second shepherd is the trainer of the dog. This approach accepts that the flock will wander; it is in its nature. Instead of trying to prevent every possible misstep, it invests in vigilance and response. This is the world of comprehensive monitoring, detailed logging, and automated remediation scripts. The dog—our collection of watchful processes and alerting systems—patrols the perimeter. It barks when something looks amiss and nips at the heels of a process straying too far from its resource limits. It doesn't prevent the error from happening in the first place, but it ensures we know about it the instant it does and often has a pre-programmed way to herd things back to safety.
The Pasture at 3 AM
The fundamental difference between these two shepherds reveals itself most clearly in the dead of night. The fence-builder sleeps soundly. Their world is predictable; a breach is a catastrophic failure of the design itself, a rare event. The dog-trainer, however, is a light sleeper. Their pager is an extension of their arm. They trust not in the perfection of the enclosure, but in the speed and intelligence of the response. Their world is dynamic, accepting of flux, but demanding of attention.
Neither approach is inherently superior, and most of us practice a blend of both. We build the strongest fences we can—enforcing strict permissions and immutable deployments—while also keeping a well-trained dog or two on hand—like a detailed audit log that can trace the ‘how’ and ‘who’ if a breach ever occurs. The art lies in knowing where to invest the effort. For a critical financial database, we build the highest fence imaginable. For a rapidly evolving development environment, we might rely more on the watchful dogs to contain the inevitable chaos of innovation. The choice isn't between control and chaos, but between different kinds of control: one that is rigid and upfront, and another that is fluid and eternal.
Notes & further reading
A few pages I came back to while writing this: