The Potter's Kiln: On Firing Systems and the Temper of Reliability
There is a moment in pottery, just before the kiln is sealed, that is pure faith. The potter has spent hours centering the clay, raising the walls, and shaping the form. But the final test, the one that determines if the vessel will hold water or shatter, is entirely out of their hands. It is a process of controlled, relentless heat. This act of firing, I’ve come to realize, is not so different from how we should test our small services and systems.
We often test our code in sterile, predictable environments—the equivalent of a potter examining a bone-dry, unfired cup. It looks finished. It has the right shape. But it is brittle and useless. The real test of a system is how it behaves under the heat of production load, unexpected failures, and degraded network conditions. This is the kiln. We can’t be inside the kiln to adjust the pot as it fires; similarly, we can’t manually correct a service as it buckles under a traffic spike. The system must have been built with the right temper to survive on its own.
The Wisdom of the Firing Schedule
A potter doesn’t just crank the kiln to maximum temperature. A rapid rise causes thermal shock and guarantees cracks. Instead, they follow a precise ‘firing schedule’—a slow ramp-up to burn away moisture, a steady climb to maturity, and sometimes a controlled cool-down. Our deployment and load-testing strategies should borrow this patience. Ramping up a new service version too quickly is its own form of thermal shock. Canaries, gradual rollouts, and shadow traffic are our firing schedule. They are the controlled application of heat that reveals weaknesses before the entire batch is committed.
And what about a failed firing? A potter opens the kiln to find a exploded piece or a glaze that ran. The response isn’t despair; it’s analysis. They examine the shards—the ‘logs’ of the process—to understand what went wrong. Was the clay too wet? Was the temperature rise too fast? We have our own shards: core dumps, error logs, and metrics from a failed deployment. The lesson from the pottery studio is to cherish these fragments. They are not evidence of failure but the only tangible guide to understanding the complex alchemy of heat, time, and material.
Ultimately, both the potter and the system builder are engaged in a quest for reliability. For the potter, it’s a vessel that doesn’t leak. For us, it’s a service that doesn’t break. This reliability isn’t a feature you add at the end; it is baked into the very fabric of the thing through rigorous, unforgiving testing. It’s earned in the kiln, not at the wheel. Our goal should be to construct systems that we can put into the fire with confidence, knowing that the careful preparation of our own ‘clay’ and ‘glaze’ will result in something strong, resilient, and beautifully fit for purpose.
Notes & further reading
A few pages I came back to while writing this: