The Keeper of the Old Sump Pump

In a dusty corner of our server room, in a rack that has seen better days, lives Marvin. Marvin isn’t a person, but a role. It’s a title bestowed upon the oldest, most reliable piece of hardware that still has a critical job to do. The current Marvin is a network-attached storage box, a model so old its web interface only renders correctly in Firefox 52. It has a single purpose: to run the script that checks the integrity of our quarterly, cold-storage backups.

The script itself is a simple bash affair. It doesn’t need to be clever. Once a week, it wakes up, mounts the dusty external drives we’ve rotated in from the fireproof safe, and performs a checksum verification on the data. It logs its start time, its end time, and a single line of output: ‘OK’ or ‘FAIL’. This log is not fed into our fancy, cloud-based log aggregator. It’s appended to a plain text file on Marvin’s tiny, spinning-platter boot drive. Reading that file requires a direct SSH connection, a password stored in a sealed envelope in the CEO’s desk, and a strange sense of ceremony.

The Ritual of the Green Light

This is where the tradition comes in. Every Monday morning, one of the senior engineers—we rotate the duty—walks over to Marvin’s rack. They don’t check the file remotely. The ritual demands a physical presence. They peer through the grille at the front, looking for a specific green LED on the chassis. If it’s blinking steadily, the weekly verification passed. The log file is, at this point, almost a formality. The light is the signal.

There’s no alert if the light isn’t blinking. There’s no pager duty for Marvin. Its reliability is assumed, and its failure would be noticed through this weekly, human check. This feels antediluvian in an age of automated dashboards and AI-driven anomaly detection. It’s our version of a sump pump in a basement: you don’t think about it until you need it, and you only know it’s failed when the floor gets wet. But the act of looking, the physical ritual, embeds the system’s existence into our team’s consciousness in a way a Slack notification never could.

We talk about backups in terms of RPO and RTO, but we rarely talk about the rituals that give us confidence in them. The cold, hard drives in the safe are only as good as our belief that they work. Marvin and its green light provide that belief. It’s a tangible, almost mundane interaction with a system that exists solely for a catastrophe we hope never arrives. It connects us to the stark reality of recovery, a process that will be messy and human, not a sleek, one-click cloud console operation.

New engineers often suggest replacing Marvin. They point to its age, its inefficiency. They propose elegant serverless functions or a robust Kubernetes job. We listen, and then we explain the tradition. Marvin isn’t just a machine; it’s the keeper of a very specific, very important kind of boredom. Its simplicity is its armor. Its ritual is its heartbeat. And as long as that little green light keeps blinking every Monday, we can sleep a little easier, knowing that in our quietest corner, a silent guardian is doing its one, profoundly boring, and absolutely vital job.

Notes & further reading

A few pages I came back to while writing this: