The Unblinking Eye of the Old Water Tank
It was the silence that woke me. Not a crash, not an alarm, but the profound, unnatural absence of a specific, low hum that had been the soundtrack to my life for a decade. I lay there in the dark, my mind trying to parse the void. The server rack in the next room was quiet, its cooling fans stilled. The power was out.
My first thought wasn't for the servers themselves. They were modern, coddled things with battery backups that would give them a graceful few minutes to shut down. My feet hit the cold floor, and I was moving not towards the rack, but towards the basement stairs. My concern was for the old beast in the corner: the creaky, custom-built FreeNAS box that served as our primary backup target. Its array of spinning disks had no graceful shutdown. A sudden power loss was a violent act, a gamble with every platter.
I found it in the pitch black, its own status lights extinguished. I placed a hand on its warm, brushed metal case. It felt like a sleeping animal. In that darkness, the abstraction of "backups" vanished. This was no longer a service or a piece of boring technology; it was a physical thing, a vessel. Inside it was the only copy of a project we had worked on for two years, a digital artifact too large and too transient for the cloud. It was the unblinking eye that had witnessed every change, every mistake, every save. And now it was blind.
The next hour was spent in a quiet, methodical panic. I found a flashlight, located the ancient UPS it was supposedly plugged into—its battery long since deceased, a failure in my own complacency—and waited. When the lights finally flickered back on, I held my breath and pressed the power button. The beast groaned to life, its disks spinning up in a staggered chorus that sounded, to my anxious ears, like a death rattle. The console scrolled through its self-check. For a full five minutes, it was a black box containing either salvation or ruin.
Then, the login prompt appeared. A quick check confirmed the ZFS pool was healthy, its checksums all valid. The data was intact. The relief was physical, a wave of warmth that washed away the tension in my shoulders. I hadn’t just recovered data; I had re-learned a fundamental truth. Reliability isn't a feature you enable. It's a physical property, a covenant between you and the machine. It’s the choice to replace the dead battery in the UPS, to listen for the hum in the night, to remember that the most critical services often live in the oldest, quietest corners, asking for nothing but a little constant care, and offering everything in return.
That old water tank of a server, unglamorous and ignored, had done its one job perfectly. It had held the water, even when the well pump stopped. I gave its warm case a silent, grateful pat, a gesture for a thing that had just taught me that the most important blink is the one that never happens.
Notes & further reading
A few pages I came back to while writing this: