The Ferryman and the Bridge: On Two Kinds of Data Migration
Last quarter, we faced the quiet, inevitable task that every system outgrows: the data migration. A core service, the old workhorse of our infrastructure, needed to move its main dataset from one storage system to another. The data wasn’t vast by modern standards, but it was the bedrock of our operations—a record of every transaction, every state change, for years. The risk wasn’t catastrophic failure, but the subtle, erosive kind: corruption, inconsistency, the slow bleed of trust. As we planned, two distinct philosophies emerged, and I came to think of them as the Ferryman and the Bridge.
The Ferryman approach was championed first. It argued for a clean break. We would quiesce the service at a predetermined, quiet hour—our own midnight. We’d take a final backup, a perfect snapshot of the old world. Then, a script, meticulously tested, would act as the Ferryman, rowing the data across the silent river from the old storage to the new. The service would remain down, dark, until the ferry reached the other side. Only then would we bring it back up, pointing to the new datastore. The appeal was its clarity, its ritual finality. There was a single moment of migration, a single event to succeed or fail. The downside, of course, was the required downtime. The ferry only runs so fast, and the river only so narrow.
The Appeal of the Permanent Crossing
The Bridge proposal was different. It was an argument for continuity. Instead of a single journey, we would build a structure that allowed for a continuous, two-way flow. We would first bring the new storage system online, a new shore beside the old. Then, we’d modify the service to write every piece of data to both systems simultaneously. For a period, the two datastores would live in sync, a double-entry ledger of our activities. Once we were confident in the new system’s stability, we would begin reading exclusively from it, while still writing to both. Finally, after a suitable observation period, we would sever the connection to the old world. The old storage would become a read-only monument, a preserved bank, before being decommissioned. The service would never know a moment of outage.
The debate between these two approaches is a classic tension in our line of work. The Ferryman is simple, understandable, and possesses a kind of austere beauty. Its success is binary, its rollback plan is as simple as reverting to the pre-migration backup. But it demands a sacrifice: the service’s availability. The Bridge is complex, requiring more code, more coordination, and a more sophisticated understanding of data consistency. It trades the simplicity of a single event for the continuous, humming effort of maintaining two truths at once. Its failure modes are subtler—what if the sync drifts?—but its primary virtue is that it never asks the user to wait on the riverbank.
We, after much deliberation, built the Bridge. The migration took weeks from start to finish, a long, drawn-out process of validation and careful observation. There was no dramatic midnight cutover, no single moment to toast with coffee. The success was invisible, marked only by the quiet deletion of the dual-write code and the final shutdown of the old database. It was the opposite of exciting, which, in our world, is the highest compliment you can pay to an operation. The Ferryman’s single, heroic journey is a compelling story. But the Bridge, silent and always open, is the infrastructure you can truly build upon.
Notes & further reading
A few pages I came back to while writing this: