The Potter's Clay: On the Temperament of Stateful Data

I’ve been spending my weekends at a friend’s pottery studio, learning to center clay on the wheel. It’s a humbling exercise in applied physics and patience. You apply pressure, but it must be even and constant. Too much force in one spot, and the entire mass wobbles, loses its axis, and collapses into a sad, wet lump. The clay, I was told, has a ‘memory’. It remembers every uneven touch, every hesitation, and it will reveal those flaws later, as the piece dries and fires. It struck me that this is a near-perfect analogy for managing stateful services and their data.

The Memory in the Medium

Our databases, message queues, and file stores are not inert buckets. They are clay. Every transaction, every write, every configuration change is a touch that shapes the medium. A rushed migration is a thumb pressed too hard. An inconsistent backup strategy is an uneven base. The system remembers. We might not see the wobble immediately—the service may keep spinning—but the memory is there, waiting to manifest as latency, corruption, or a catastrophic failure during a restore, the equivalent of the kiln.

Potters have a concept called ‘wedging’, the process of kneading the clay to remove air bubbles and ensure a uniform consistency. It’s boring, repetitive, and absolutely critical. A single hidden bubble can cause a finished vessel to explode in the heat. Our operational ‘wedging’ is the boring, reliable work we often skip: consistent log rotation that doesn’t fill a disk, regular integrity checks on backups, the gradual aging-out of old data formats. We skip it because the clay seems fine. It spins. But we are introducing bubbles.

The most profound lesson, however, is in the handling of flaws. A potter will sometimes spot a hairline crack as a piece dries (the ‘greenware’ stage). At this point, they can use ‘slip’—a slurry of clay and water—to mend it. Once the piece is fired into ‘bisqueware’, that crack is permanent; you can only glaze over it. This is the stark difference between a live repair on a running system and a post-mortem fix after an outage. Our ‘slip’ is the ability to apply a hotfix, to run a data correction script while the service is live, to re-sync a replica. It requires acknowledging the flaw while the system is still malleable. Once the ‘fire’ of a public incident hits, the fracture is set.

I now look at our aging Postgres cluster differently. It is not a machine; it is a large, slowly-turning wheel of clay, shaped by three years of features, tech debt, and ad-hoc patches. My job is not to be its mechanic, but its porter. To apply steady, even pressure. To wedge out the bubbles of complacency before the next scaling event turns up the heat. And to have the courage to apply the ‘slip’ of a corrective migration now, rather than just hoping the glaze of a load balancer will hide the crack later. The medium remembers, and it is our duty to shape memories worth keeping.

Notes & further reading

A few pages I came back to while writing this: