The Lamplighter's Dimming Flame: On the Allure and Abyss of the Single Point of Truth

In the systems we build, there’s a siren song we all hear at some point: the promise of the Single Point of Truth. It’s an idea so elegant, so seductive, it feels like the pinnacle of good design. Consolidate everything into one authoritative source—one database, one configuration file, one master service. Imagine the simplicity! No more conflicting data, no more synchronization headaches, no more wondering which version is correct. The SPoT becomes our digital lamplighter, a solitary, reliable flame casting light across the entire town of our infrastructure. We build our services to gather around it, drawing their certainty from its unwavering glow.

And for a while, it works beautifully. The system is fast, coherent, and easy to reason about. We congratulate ourselves on our architectural purity. But the received wisdom we so eagerly accepted has a critical, often unstated, dependency: it assumes a world of perfect conditions. It assumes the lamplighter never gets sick, the flame never sputters, and the night is never stormy. In other words, it assumes a reality that does not exist.

The trouble begins not as a dramatic explosion, but as a slow, creeping dimness. A network partition, a disk filling up with logs, a subtle bug in the garbage collection—any small, mundane failure that would be a hiccup in a distributed system becomes a total blackout. The single light goes out, and with it, the entire town is plunged into darkness. The very simplicity we craved becomes our single point of failure. Services that were happily dependent are now utterly paralyzed. They can’t make a decision, process a request, or often even start up, because their sole source of truth is gone.

This is the abyss hidden within the allure. The SPoT creates a system with zero redundancy. There is no backup light, no cache of yesterday’s flame, no way for a service to function, even in a degraded mode, without it. We traded resilience for a clean diagram. We mistook centralization for clarity. In our quest to eliminate the complexity of managing multiple sources, we built a system so fragile that its reliability is entirely contingent on the flawless operation of one component—a bet that any seasoned operator knows is a fool’s wager.

The true wisdom, the boring and reliable alternative, is to embrace a certain amount of distributed, managed complexity. It means building services that can tolerate ambiguity and operate with stale data. It means having fallback caches, read replicas, or even the ability to function in a read-only mode based on a recent snapshot. It’s messier on paper. It requires more thoughtful code for handling data inconsistencies. But it builds a system that can stumble without falling, a town where if the main light goes out, the shops can still open by lantern light until the lamplighter returns. The goal isn’t to avoid having a primary source, but to avoid having a single point of catastrophic failure. The most reliable flame is not the one that burns brightest alone, but the one that has been used to light others.

Notes & further reading

A few pages I came back to while writing this: